[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:35:15 dist-test-slave-rpg6 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:35:15 dist-test-slave-rpg6 krb5kdc[395](info): set up 2 sockets
Jun 20 02:35:15 dist-test-slave-rpg6 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 02:35:16.941569 388 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:35:17 dist-test-slave-rpg6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:35:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:35:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:35:17.141968 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36767
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:17.508971 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:17.509316 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:17.509406 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:17.516891 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:35:17.517023 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:17.517077 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:17.517138 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:35:17.517187 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:35:17.527911 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:17.529341 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:17.530745 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:17.544806 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:17.544787 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:17.544857 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:17.545850 408 server_base.cc:1034] running on GCE node
I20260620 02:35:17.546835 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:17.548659 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:17.549883 408 hybrid_clock.cc:648] HybridClock initialized: now 1781922917549746 us; error 137 us; skew 500 ppm
I20260620 02:35:17.550231 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:17 dist-test-slave-rpg6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:17.553745 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:35:17.555522 408 webserver.cc:460] Webserver started at http://127.0.97.62:33375/ using document root <none> and password file <none>
I20260620 02:35:17.556386 408 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:17.556524 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:17.556875 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:17.560379 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "e22e6ab9a0424a6dbf6a00c47820da7c"
format_stamp: "Formatted at 2026-06-20 02:35:17 on dist-test-slave-rpg6"
server_key: "f3be6ae94ce601ba7db5fbf39d0347af"
server_key_iv: "ea6a0b76015bff188544a695d91f563f"
server_key_version: "encryptionkey@0"
I20260620 02:35:17.561255 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "e22e6ab9a0424a6dbf6a00c47820da7c"
format_stamp: "Formatted at 2026-06-20 02:35:17 on dist-test-slave-rpg6"
server_key: "f3be6ae94ce601ba7db5fbf39d0347af"
server_key_iv: "ea6a0b76015bff188544a695d91f563f"
server_key_version: "encryptionkey@0"
I20260620 02:35:17.567600 408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:35:17.572293 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:17.573925 408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:35:17.574257 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "e22e6ab9a0424a6dbf6a00c47820da7c"
format_stamp: "Formatted at 2026-06-20 02:35:17 on dist-test-slave-rpg6"
server_key: "f3be6ae94ce601ba7db5fbf39d0347af"
server_key_iv: "ea6a0b76015bff188544a695d91f563f"
server_key_version: "encryptionkey@0"
I20260620 02:35:17.574505 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:17.590474 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:17.595445 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:17.595805 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:17.617300 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36767
I20260620 02:35:17.617337 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36767 every 8 connection(s)
I20260620 02:35:17.618850 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:35:17.623469 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 408
I20260620 02:35:17.623580 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:17.623816 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:35:17.626425 388 external_mini_cluster.cc:1428] Setting key d99440c366cc2b90579fd1d9b7296d85
I20260620 02:35:17.637372 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c: Bootstrap starting.
I20260620 02:35:17.641398 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:17.642840 475 log.cc:824] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:17.646782 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c: No bootstrap required, opened a new log
I20260620 02:35:17.656761 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22e6ab9a0424a6dbf6a00c47820da7c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36767 } }
I20260620 02:35:17.657249 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:17.657375 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22e6ab9a0424a6dbf6a00c47820da7c, State: Initialized, Role: FOLLOWER
I20260620 02:35:17.658164 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22e6ab9a0424a6dbf6a00c47820da7c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36767 } }
I20260620 02:35:17.658525 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:35:17.658674 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:35:17.658879 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [term 0 FOLLOWER]: Advancing to term 1
Jun 20 02:35:17 dist-test-slave-rpg6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:17.662041 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22e6ab9a0424a6dbf6a00c47820da7c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36767 } }
I20260620 02:35:17.662609 475 leader_election.cc:302] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e22e6ab9a0424a6dbf6a00c47820da7c; no voters:
I20260620 02:35:17.663321 475 leader_election.cc:288] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:35:17.663494 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:17.665525 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [term 1 LEADER]: Becoming Leader. State: Replica: e22e6ab9a0424a6dbf6a00c47820da7c, State: Running, Role: LEADER
I20260620 02:35:17.666182 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22e6ab9a0424a6dbf6a00c47820da7c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36767 } }
I20260620 02:35:17.666337 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:35:17.669232 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e22e6ab9a0424a6dbf6a00c47820da7c. Latest consensus state: current_term: 1 leader_uuid: "e22e6ab9a0424a6dbf6a00c47820da7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22e6ab9a0424a6dbf6a00c47820da7c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36767 } } }
I20260620 02:35:17.669675 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:17.670229 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e22e6ab9a0424a6dbf6a00c47820da7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22e6ab9a0424a6dbf6a00c47820da7c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36767 } } }
I20260620 02:35:17.670516 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:17.680635 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:17.632093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.0.97.62:36767)
0620 02:35:17.633059 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:17.633076 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:17.634524 (+ 1448us) server_negotiation.cc:407] Connection header received
0620 02:35:17.644648 (+ 10124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:17.644686 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:17.645277 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:17.645833 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:17.649262 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:17.651967 (+ 2705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:17.653999 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:17.655245 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:17.655481 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:17.663113 (+ 7632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:17.663131 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:17.663165 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:17.663231 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:17.668766 (+ 5535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:17.670411 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:17.670428 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:17.670444 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:17.670778 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:17.671592 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:17.671600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:17.671606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:17.672721 (+ 1115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:17.677098 (+ 4377us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:17.677549 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 02:35:17.678233 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":432,"threads_started":1}
I20260620 02:35:17.681182 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:35:17.686645 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:35:17.697788 485 catalog_manager.cc:1347] Generated new cluster ID: 17b3a9bc7a0e4d6da52151260b4d2e56
I20260620 02:35:17.697923 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:35:17.714716 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:35:17.716519 485 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:35:17.723923 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e22e6ab9a0424a6dbf6a00c47820da7c: Generated new TSK 0
I20260620 02:35:17.724902 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:35:17.780314 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36767
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:18.194840 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:18.195149 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:18.195259 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:18.202968 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:18.203110 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:18.203240 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:35:18.214265 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:18.215677 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:18.217257 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:18.231590 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:18.231761 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:18.232303 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:18.238447 502 server_base.cc:1034] running on GCE node
I20260620 02:35:18.239888 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:18.241568 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:18.242983 502 hybrid_clock.cc:648] HybridClock initialized: now 1781922918242897 us; error 53 us; skew 500 ppm
I20260620 02:35:18.243582 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:18 dist-test-slave-rpg6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:18.248417 502 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:35:18.250200 502 webserver.cc:460] Webserver started at http://127.0.97.1:32941/ using document root <none> and password file <none>
I20260620 02:35:18.251063 502 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:18.251219 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:18.251572 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:18.254859 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "810595b7d42348e993d72edd6e3453a5"
format_stamp: "Formatted at 2026-06-20 02:35:18 on dist-test-slave-rpg6"
server_key: "1869c852ead63f43e00c38e6851561f8"
server_key_iv: "50588f00c3d24c9016b08f355f1e54d6"
server_key_version: "encryptionkey@0"
I20260620 02:35:18.255772 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "810595b7d42348e993d72edd6e3453a5"
format_stamp: "Formatted at 2026-06-20 02:35:18 on dist-test-slave-rpg6"
server_key: "1869c852ead63f43e00c38e6851561f8"
server_key_iv: "50588f00c3d24c9016b08f355f1e54d6"
server_key_version: "encryptionkey@0"
I20260620 02:35:18.263455 502 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260620 02:35:18.268364 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:18.270444 502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:35:18.270682 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "810595b7d42348e993d72edd6e3453a5"
format_stamp: "Formatted at 2026-06-20 02:35:18 on dist-test-slave-rpg6"
server_key: "1869c852ead63f43e00c38e6851561f8"
server_key_iv: "50588f00c3d24c9016b08f355f1e54d6"
server_key_version: "encryptionkey@0"
I20260620 02:35:18.270953 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:18.295985 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:18.302065 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:18.302414 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:18.303813 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:18.305500 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:18.305583 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:18.305682 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:18.305727 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:18.347291 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38705
I20260620 02:35:18.347337 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38705 every 8 connection(s)
I20260620 02:35:18.348779 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:35:18.353495 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 502
I20260620 02:35:18.353730 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:35:18.354357 388 external_mini_cluster.cc:1428] Setting key 3243e278c0fc1569ca2612ccaf3f4bd2
Jun 20 02:35:18 dist-test-slave-rpg6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:35:18.374889 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:18.351792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34531 (local address 127.0.97.62:36767)
0620 02:35:18.352365 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:18.352369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:18.353758 (+ 1389us) server_negotiation.cc:407] Connection header received
0620 02:35:18.355972 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:18.355979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:18.356078 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:18.356199 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:18.357948 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:18.358697 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:18.360545 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:18.360957 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:18.361111 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:18.366564 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:18.366575 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:18.366583 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:18.366692 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:18.371047 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:18.372180 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:18.372185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:18.372188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:18.372318 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:18.372793 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:18.372797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:18.372799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:18.373393 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:18.373514 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:18.374315 (+ 801us) server_negotiation.cc:299] Negotiation successful
0620 02:35:18.374544 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":252,"threads_started":1}
I20260620 02:35:18.375948 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:18.352300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36767 (local address 127.0.97.1:34531)
0620 02:35:18.353073 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:18.353612 (+ 539us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:18.355629 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:18.356470 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:18.356480 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:18.357075 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:18.357725 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:18.357745 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:18.358889 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:18.358893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:18.360378 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:18.360393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:18.361218 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:18.361226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:18.361633 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:18.362448 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:18.362490 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:18.366172 (+ 3682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:18.371304 (+ 5132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:18.371313 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:18.371331 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:18.372029 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:18.372453 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:18.372457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:18.372460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:18.372663 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:18.373527 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:18.373534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:18.373911 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 02:35:18.374158 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 02:35:18.374606 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":307,"threads_started":1}
I20260620 02:35:18.378253 630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36767
I20260620 02:35:18.378650 630 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:18.379593 630 heartbeater.cc:511] Master 127.0.97.62:36767 requested a full tablet report, sending...
I20260620 02:35:18.382215 440 ts_manager.cc:194] Registered new tserver with Master: 810595b7d42348e993d72edd6e3453a5 (127.0.97.1:38705)
I20260620 02:35:18.383796 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:35:18.412604 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36767
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:18.783242 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:18.783630 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:18.783800 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:18.792155 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:18.792333 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:18.792519 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:35:18.802922 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:18.804447 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:18.805912 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:18.821849 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:18.821933 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:18.821906 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:18.823313 636 server_base.cc:1034] running on GCE node
I20260620 02:35:18.824131 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:18.825222 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:18.826490 636 hybrid_clock.cc:648] HybridClock initialized: now 1781922918826418 us; error 70 us; skew 500 ppm
I20260620 02:35:18.826889 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:18 dist-test-slave-rpg6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:18.830616 636 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:35:18.832445 636 webserver.cc:460] Webserver started at http://127.0.97.2:33109/ using document root <none> and password file <none>
I20260620 02:35:18.833230 636 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:18.833379 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:18.833714 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:18.837078 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "c09acdeadcda4e80a29c9d9c09fcd2c2"
format_stamp: "Formatted at 2026-06-20 02:35:18 on dist-test-slave-rpg6"
server_key: "4d66449af9d82fe7a3d6fc3d589bafd1"
server_key_iv: "83da4686ff705d0f2aa3e85ccea16f79"
server_key_version: "encryptionkey@0"
I20260620 02:35:18.838002 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c09acdeadcda4e80a29c9d9c09fcd2c2"
format_stamp: "Formatted at 2026-06-20 02:35:18 on dist-test-slave-rpg6"
server_key: "4d66449af9d82fe7a3d6fc3d589bafd1"
server_key_iv: "83da4686ff705d0f2aa3e85ccea16f79"
server_key_version: "encryptionkey@0"
I20260620 02:35:18.844882 636 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:35:18.849119 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:18.850862 636 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260620 02:35:18.851107 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "c09acdeadcda4e80a29c9d9c09fcd2c2"
format_stamp: "Formatted at 2026-06-20 02:35:18 on dist-test-slave-rpg6"
server_key: "4d66449af9d82fe7a3d6fc3d589bafd1"
server_key_iv: "83da4686ff705d0f2aa3e85ccea16f79"
server_key_version: "encryptionkey@0"
I20260620 02:35:18.851359 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:18.877207 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:18.882246 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:18.882561 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:18.883597 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:18.885547 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:18.885655 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260620 02:35:18.885798 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:18.885867 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:18.925936 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38889
I20260620 02:35:18.926028 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38889 every 8 connection(s)
I20260620 02:35:18.927357 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:35:18.932914 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 636
I20260620 02:35:18.933162 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:35:18.933585 388 external_mini_cluster.cc:1428] Setting key 674c6eb0d3f205cd89fcd61772b185fb
Jun 20 02:35:18 dist-test-slave-rpg6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:35:18.950529 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:18.930646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48167 (local address 127.0.97.62:36767)
0620 02:35:18.931105 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:18.931110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:18.932737 (+ 1627us) server_negotiation.cc:407] Connection header received
0620 02:35:18.934946 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:18.934952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:18.935037 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:18.935167 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:18.936938 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:18.937678 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:18.939353 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:18.939803 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:18.940005 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:18.944244 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:18.944250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:18.944254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:18.944291 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:18.947047 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:18.947978 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:18.947983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:18.947985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:18.948125 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:18.948603 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:18.948607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:18.948610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:18.948897 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:18.949061 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:18.949889 (+ 828us) server_negotiation.cc:299] Negotiation successful
0620 02:35:18.950149 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":248,"threads_started":1}
I20260620 02:35:18.951712 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:18.931257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36767 (local address 127.0.97.2:48167)
0620 02:35:18.932556 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:18.932597 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:18.934663 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:18.935454 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:18.935464 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:18.936107 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:18.936752 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:18.936770 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:18.937858 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:18.937862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:18.939187 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:18.939200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:18.940131 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:18.940139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:18.940400 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:18.941222 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:18.941250 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:18.943958 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:18.947244 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:18.947255 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:18.947282 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:18.947787 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:18.948250 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:18.948255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:18.948259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:18.948474 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:18.949053 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:18.949062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:18.949483 (+ 421us) client_negotiation.cc:769] Sending connection context
0620 02:35:18.949795 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 02:35:18.950408 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":427,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:35:18.953658 764 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36767
I20260620 02:35:18.954279 764 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:18.955250 764 heartbeater.cc:511] Master 127.0.97.62:36767 requested a full tablet report, sending...
I20260620 02:35:18.957183 440 ts_manager.cc:194] Registered new tserver with Master: c09acdeadcda4e80a29c9d9c09fcd2c2 (127.0.97.2:38889)
I20260620 02:35:18.958062 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:35:18.979924 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36767
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:19.357645 770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:19.357954 770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:19.358052 770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:19.366163 770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:19.366314 770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:19.366434 770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:35:19.376993 770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:19.378748 770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:19.380311 770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 02:35:19.388477 630 heartbeater.cc:503] Master 127.0.97.62:36767 was elected leader, sending a full tablet report...
W20260620 02:35:19.397927 775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:19.398037 778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:19.398126 776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:19.398766 770 server_base.cc:1034] running on GCE node
I20260620 02:35:19.399534 770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:19.400715 770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:19.401991 770 hybrid_clock.cc:648] HybridClock initialized: now 1781922919401943 us; error 52 us; skew 500 ppm
I20260620 02:35:19.402474 770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:19 dist-test-slave-rpg6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:19.406244 770 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:35:19.408658 770 webserver.cc:460] Webserver started at http://127.0.97.3:40435/ using document root <none> and password file <none>
I20260620 02:35:19.409554 770 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:19.409713 770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:19.410048 770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:19.413465 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "25d9627b60fb41b88f30c206e426d5fb"
format_stamp: "Formatted at 2026-06-20 02:35:19 on dist-test-slave-rpg6"
server_key: "c2ea49981e7cccff7d19829433ebae1d"
server_key_iv: "aac40775cc213a900857451b7aaa5132"
server_key_version: "encryptionkey@0"
I20260620 02:35:19.414484 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "25d9627b60fb41b88f30c206e426d5fb"
format_stamp: "Formatted at 2026-06-20 02:35:19 on dist-test-slave-rpg6"
server_key: "c2ea49981e7cccff7d19829433ebae1d"
server_key_iv: "aac40775cc213a900857451b7aaa5132"
server_key_version: "encryptionkey@0"
I20260620 02:35:19.420861 770 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 02:35:19.424867 785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:19.427273 770 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:35:19.427595 770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "25d9627b60fb41b88f30c206e426d5fb"
format_stamp: "Formatted at 2026-06-20 02:35:19 on dist-test-slave-rpg6"
server_key: "c2ea49981e7cccff7d19829433ebae1d"
server_key_iv: "aac40775cc213a900857451b7aaa5132"
server_key_version: "encryptionkey@0"
I20260620 02:35:19.427986 770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:19.447508 770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:19.453624 770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:19.453903 770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:19.455294 770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:19.456972 770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:19.457062 770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:19.457154 770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:19.457197 770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:19.500631 770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34361
I20260620 02:35:19.500633 897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34361 every 8 connection(s)
I20260620 02:35:19.502230 770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:35:19.511348 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 770
I20260620 02:35:19.511575 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:35:19.512126 388 external_mini_cluster.cc:1428] Setting key e8c063b23456e6d55733a8be19c18437
Jun 20 02:35:19 dist-test-slave-rpg6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:19.525390 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:19.505491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36123 (local address 127.0.97.62:36767)
0620 02:35:19.505975 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:19.505980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:19.507372 (+ 1392us) server_negotiation.cc:407] Connection header received
0620 02:35:19.509427 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:19.509433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:19.509529 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:19.509653 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:19.511378 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.512463 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.514411 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.514861 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.515041 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:19.519255 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:19.519261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:19.519264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:19.519303 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:19.522228 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.523075 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.523080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.523082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.523209 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.523663 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.523666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.523668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.524025 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:19.524145 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:19.524883 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 02:35:19.525064 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":260,"threads_started":1}
I20260620 02:35:19.526413 901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:19.505934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36767 (local address 127.0.97.3:36123)
0620 02:35:19.507176 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:19.507226 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:19.509089 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:19.509935 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:19.509945 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:19.510499 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:19.511159 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:19.511178 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.512647 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.512651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:19.514256 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:19.514270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.515219 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.515226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:19.515456 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:19.516525 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:19.516552 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:19.518963 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:19.522421 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:19.522430 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:19.522446 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:19.522940 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:19.523353 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:19.523356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:19.523359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:19.523547 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:19.524169 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:19.524176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:19.524527 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 02:35:19.524780 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 02:35:19.525234 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":775,"threads_started":1}
I20260620 02:35:19.528182 898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36767
I20260620 02:35:19.528499 898 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:19.529397 898 heartbeater.cc:511] Master 127.0.97.62:36767 requested a full tablet report, sending...
I20260620 02:35:19.530892 440 ts_manager.cc:194] Registered new tserver with Master: 25d9627b60fb41b88f30c206e426d5fb (127.0.97.3:34361)
I20260620 02:35:19.531770 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36123
I20260620 02:35:19.542915 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:35:19 dist-test-slave-rpg6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922919, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:35:19 dist-test-slave-rpg6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922919, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:35:19.599498 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:19.577913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.0.97.1:38705)
0620 02:35:19.578904 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:19.578913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:19.578973 (+ 60us) server_negotiation.cc:407] Connection header received
0620 02:35:19.579215 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:19.579223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:19.579575 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:19.579748 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:19.581407 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.582587 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.584457 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.585117 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.585261 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:19.591334 (+ 6073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:19.591350 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:19.591366 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:19.591411 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:19.595194 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.596771 (+ 1577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.596781 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.596784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.596962 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.597713 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.597718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.597720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.598145 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:19.598426 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:19.598837 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 02:35:19.599056 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":605,"threads_started":1}
W20260620 02:35:19.600952 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41930
Jun 20 02:35:19 dist-test-slave-rpg6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:35:19 dist-test-slave-rpg6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:35:19.650434 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:19.633621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.0.97.1:38705)
0620 02:35:19.633833 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:19.633842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:19.634038 (+ 196us) server_negotiation.cc:407] Connection header received
0620 02:35:19.634258 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:19.634277 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:19.634417 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:19.634536 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:19.635934 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.636637 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.637937 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.638390 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.638534 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:19.643256 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:19.643263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:19.643266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:19.643311 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:19.646662 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.647480 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.647485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.647488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.647639 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.648592 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.648598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.648601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.649069 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:19.649368 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:19.649768 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 02:35:19.650112 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:35:19.652154 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 502
I20260620 02:35:19.666985 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:19.653204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.0.97.1:38705)
0620 02:35:19.653446 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:19.653452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:19.653647 (+ 195us) server_negotiation.cc:407] Connection header received
0620 02:35:19.653794 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:19.653798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:19.653857 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:19.653935 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:19.655894 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.656723 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.658135 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.658639 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.658760 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:19.661304 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:19.661309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:19.661313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:19.661356 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:19.664129 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.664862 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.664866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.664868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.665003 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.665472 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.665475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.665478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.665776 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:19.665965 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:19.666298 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 02:35:19.666479 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 02:35:19.667908 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:19.752555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 502
I20260620 02:35:19.770417 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 636
Jun 20 02:35:19 dist-test-slave-rpg6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:35:19.788127 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:19.771353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.0.97.2:38889)
0620 02:35:19.771932 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:19.771942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:19.771979 (+ 37us) server_negotiation.cc:407] Connection header received
0620 02:35:19.772042 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:19.772048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:19.772340 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:19.772487 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:19.773711 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.774862 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.776347 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.776928 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.777129 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:19.781377 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:19.781394 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:19.781409 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:19.781462 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:19.784849 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.785669 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.785677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.785680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.785847 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.786294 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.786304 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.786307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.786696 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:19.787045 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:19.787357 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 02:35:19.787644 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":305,"threads_started":1}
I20260620 02:35:19.789623 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:19.874330 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 636
I20260620 02:35:19.898509 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 770
Jun 20 02:35:19 dist-test-slave-rpg6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:35:19.917661 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:19.899699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.0.97.3:34361)
0620 02:35:19.900424 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:19.900431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:19.900460 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:35:19.900522 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:19.900529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:19.900759 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:19.900910 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:19.902163 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.903269 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.904823 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:19.905436 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:19.905546 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:19.910993 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:19.911009 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:19.911028 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:19.911071 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:19.914415 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.915294 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.915305 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.915308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.915474 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:19.915947 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:19.915951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:19.915953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:19.916358 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:19.916599 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:19.916965 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 02:35:19.917265 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":467,"threads_started":1}
I20260620 02:35:19.919094 893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:20.006603 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 770
I20260620 02:35:20.025043 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 408
I20260620 02:35:20.026417 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:20.134774 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 408
2026-06-20T02:35:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4354 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:35:20 dist-test-slave-rpg6 krb5kdc[920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:35:20 dist-test-slave-rpg6 krb5kdc[920](info): set up 2 sockets
Jun 20 02:35:20 dist-test-slave-rpg6 krb5kdc[920](info): commencing operation
krb5kdc: starting...
W20260620 02:35:21.232024 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:35:21 dist-test-slave-rpg6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:35:21.424782 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39233
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:21.776333 933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:21.776683 933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:21.776851 933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:21.784880 933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:35:21.785028 933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:21.785108 933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:21.785218 933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:35:21.785286 933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:35:21.796443 933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:21.797889 933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:21.799345 933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:21.813838 938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:21.813866 941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:21.813836 939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:21.814917 933 server_base.cc:1034] running on GCE node
I20260620 02:35:21.816030 933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:21.817636 933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:21.818917 933 hybrid_clock.cc:648] HybridClock initialized: now 1781922921818854 us; error 49 us; skew 500 ppm
I20260620 02:35:21.819384 933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:21 dist-test-slave-rpg6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:21.826900 933 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:35:21.828910 933 webserver.cc:460] Webserver started at http://127.0.97.62:35833/ using document root <none> and password file <none>
I20260620 02:35:21.829728 933 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:21.829882 933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:21.830221 933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:21.833702 933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "8ad9b3fc68944225aa7d746634ad9aa4"
format_stamp: "Formatted at 2026-06-20 02:35:21 on dist-test-slave-rpg6"
server_key: "9d09f833aa71685423d25ea1d2c69396"
server_key_iv: "802b00e442d649f26e15ba66b73b43a4"
server_key_version: "encryptionkey@0"
I20260620 02:35:21.834714 933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ad9b3fc68944225aa7d746634ad9aa4"
format_stamp: "Formatted at 2026-06-20 02:35:21 on dist-test-slave-rpg6"
server_key: "9d09f833aa71685423d25ea1d2c69396"
server_key_iv: "802b00e442d649f26e15ba66b73b43a4"
server_key_version: "encryptionkey@0"
I20260620 02:35:21.841102 933 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:35:21.845419 948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:21.847085 933 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:35:21.847327 933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "8ad9b3fc68944225aa7d746634ad9aa4"
format_stamp: "Formatted at 2026-06-20 02:35:21 on dist-test-slave-rpg6"
server_key: "9d09f833aa71685423d25ea1d2c69396"
server_key_iv: "802b00e442d649f26e15ba66b73b43a4"
server_key_version: "encryptionkey@0"
I20260620 02:35:21.847561 933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:21.868338 933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:21.873927 933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:21.874305 933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:21.899268 933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39233
I20260620 02:35:21.899289 999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39233 every 8 connection(s)
I20260620 02:35:21.901037 933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:35:21.905536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 933
I20260620 02:35:21.905675 1000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:21.905809 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:35:21.906580 388 external_mini_cluster.cc:1428] Setting key b723d219805b427e09f8748bf8ecb9bc
I20260620 02:35:21.920126 1000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4: Bootstrap starting.
Jun 20 02:35:21 dist-test-slave-rpg6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:21.923930 1000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:21.925438 1000 log.cc:824] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:21.929162 1000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4: No bootstrap required, opened a new log
I20260620 02:35:21.934557 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:21.909221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.0.97.62:39233)
0620 02:35:21.910125 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:21.910138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:21.910189 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:35:21.912847 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:21.912881 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:21.913413 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:21.913944 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:21.915571 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:21.917063 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:21.918458 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:21.919283 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:21.919514 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:21.924423 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:21.924437 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:21.924468 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:21.924531 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:21.929281 (+ 4750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:21.930379 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:21.930388 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:21.930397 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:21.930594 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:21.931157 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:21.931161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:21.931163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:21.931819 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:21.932139 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:21.932792 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 02:35:21.933353 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":432,"threads_started":1}
I20260620 02:35:21.938484 1000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad9b3fc68944225aa7d746634ad9aa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39233 } }
I20260620 02:35:21.938916 1000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:21.939024 1000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad9b3fc68944225aa7d746634ad9aa4, State: Initialized, Role: FOLLOWER
I20260620 02:35:21.939603 1000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad9b3fc68944225aa7d746634ad9aa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39233 } }
I20260620 02:35:21.939918 1000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:35:21.940080 1000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:35:21.940261 1000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:21.942858 1000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad9b3fc68944225aa7d746634ad9aa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39233 } }
I20260620 02:35:21.943459 1000 leader_election.cc:302] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ad9b3fc68944225aa7d746634ad9aa4; no voters:
I20260620 02:35:21.944198 1000 leader_election.cc:288] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:35:21.944322 1005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:21.945099 1005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad9b3fc68944225aa7d746634ad9aa4, State: Running, Role: LEADER
I20260620 02:35:21.945703 1005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad9b3fc68944225aa7d746634ad9aa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39233 } }
I20260620 02:35:21.946768 1000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:35:21.949208 1007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ad9b3fc68944225aa7d746634ad9aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad9b3fc68944225aa7d746634ad9aa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39233 } } }
I20260620 02:35:21.949334 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ad9b3fc68944225aa7d746634ad9aa4. Latest consensus state: current_term: 1 leader_uuid: "8ad9b3fc68944225aa7d746634ad9aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad9b3fc68944225aa7d746634ad9aa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39233 } } }
I20260620 02:35:21.949637 1007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:21.950433 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:21.951737 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:35:21.957947 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:35:21.968052 1012 catalog_manager.cc:1347] Generated new cluster ID: 58ecfd99a2274917a314878c18040e90
I20260620 02:35:21.968241 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:35:21.999116 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:35:22.001331 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:35:22.017177 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ad9b3fc68944225aa7d746634ad9aa4: Generated new TSK 0
I20260620 02:35:22.018400 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:35:22.085378 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39233
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:22.459424 1026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:22.459748 1026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:22.459930 1026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:22.467983 1026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:22.468112 1026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:22.468297 1026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:35:22.486922 1026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39233
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:22.488894 1026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:22.490329 1026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:22.505225 1034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:22.505242 1032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:22.505183 1031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:22.506763 1026 server_base.cc:1034] running on GCE node
I20260620 02:35:22.507539 1026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:22.508724 1026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:22.510015 1026 hybrid_clock.cc:648] HybridClock initialized: now 1781922922509967 us; error 81 us; skew 500 ppm
I20260620 02:35:22.510445 1026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:22 dist-test-slave-rpg6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:22.514307 1026 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:35:22.516278 1026 webserver.cc:460] Webserver started at http://127.0.97.1:41595/ using document root <none> and password file <none>
I20260620 02:35:22.517113 1026 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:22.517274 1026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:22.517620 1026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:22.520921 1026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "6398e80f43a54163bc2ed589b126a539"
format_stamp: "Formatted at 2026-06-20 02:35:22 on dist-test-slave-rpg6"
server_key: "32fd9db18236117a9d0d9e7a9a0aef1b"
server_key_iv: "f3c02c229c9e0f37500e109ed4ba3842"
server_key_version: "encryptionkey@0"
I20260620 02:35:22.521842 1026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6398e80f43a54163bc2ed589b126a539"
format_stamp: "Formatted at 2026-06-20 02:35:22 on dist-test-slave-rpg6"
server_key: "32fd9db18236117a9d0d9e7a9a0aef1b"
server_key_iv: "f3c02c229c9e0f37500e109ed4ba3842"
server_key_version: "encryptionkey@0"
I20260620 02:35:22.528944 1026 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 02:35:22.533128 1041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:22.534952 1026 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:35:22.535208 1026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "6398e80f43a54163bc2ed589b126a539"
format_stamp: "Formatted at 2026-06-20 02:35:22 on dist-test-slave-rpg6"
server_key: "32fd9db18236117a9d0d9e7a9a0aef1b"
server_key_iv: "f3c02c229c9e0f37500e109ed4ba3842"
server_key_version: "encryptionkey@0"
I20260620 02:35:22.535466 1026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:22.550552 1026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:22.556341 1026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:22.556684 1026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:22.557802 1026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:22.559337 1026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:22.559448 1026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:22.559607 1026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:22.559685 1026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:22.597172 1026 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40479
I20260620 02:35:22.597187 1153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40479 every 8 connection(s)
I20260620 02:35:22.598489 1026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:35:22.608970 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 1026
I20260620 02:35:22.609293 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:35:22.610028 388 external_mini_cluster.cc:1428] Setting key 18d7b79ba81c3b50b727b450b020c531
Jun 20 02:35:22 dist-test-slave-rpg6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:22.622995 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:22.601238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51719 (local address 127.0.97.62:39233)
0620 02:35:22.601750 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:22.601755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:22.602913 (+ 1158us) server_negotiation.cc:407] Connection header received
0620 02:35:22.604984 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:22.604990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:22.605087 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:22.605231 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:22.607024 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:22.607788 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:22.609728 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:22.610180 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:22.610321 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:22.614775 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:22.614785 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:22.614792 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:22.614845 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:22.618517 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:22.619718 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:22.619723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:22.619725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:22.619960 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:22.620617 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:22.620624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:22.620626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:22.621209 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:22.621390 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:22.622503 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0620 02:35:22.622735 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":281,"threads_started":1}
I20260620 02:35:22.624652 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:22.601813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39233 (local address 127.0.97.1:51719)
0620 02:35:22.602720 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:22.602767 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:22.604651 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:22.605505 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:22.605515 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:22.606106 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:22.606805 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:22.606826 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:22.608022 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:22.608026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:22.609560 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:22.609573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:22.610437 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:22.610444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:22.610805 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:22.611615 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:22.611640 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:22.614468 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:22.618752 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:22.618766 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:22.618791 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:22.619536 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:22.620083 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:22.620091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:22.620098 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:22.620379 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:22.621425 (+ 1046us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:22.621437 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:22.621977 (+ 540us) client_negotiation.cc:769] Sending connection context
0620 02:35:22.622332 (+ 355us) client_negotiation.cc:240] Negotiation successful
0620 02:35:22.622990 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":474,"threads_started":1}
I20260620 02:35:22.626459 1154 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39233
I20260620 02:35:22.626839 1154 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:22.627667 1154 heartbeater.cc:511] Master 127.0.97.62:39233 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:35:22.630270 965 ts_manager.cc:194] Registered new tserver with Master: 6398e80f43a54163bc2ed589b126a539 (127.0.97.1:40479)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:35:22.632103 965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:35:22.667781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39233
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:23.027550 1160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:23.027956 1160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:23.028070 1160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:23.035977 1160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:23.036096 1160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:23.036295 1160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:35:23.047432 1160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39233
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:23.048980 1160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:23.050366 1160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:23.066658 1166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:23.066576 1165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:23.066977 1168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:23.067440 1160 server_base.cc:1034] running on GCE node
I20260620 02:35:23.068163 1160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:23.069398 1160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:23.072836 1160 hybrid_clock.cc:648] HybridClock initialized: now 1781922923072744 us; error 83 us; skew 500 ppm
I20260620 02:35:23.073318 1160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:23 dist-test-slave-rpg6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:23.077674 1160 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:35:23.079762 1160 webserver.cc:460] Webserver started at http://127.0.97.2:38211/ using document root <none> and password file <none>
I20260620 02:35:23.080593 1160 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:23.080765 1160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:23.081108 1160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:23.084411 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "277fa73821eb4e3384d1f8bfb09ea599"
format_stamp: "Formatted at 2026-06-20 02:35:23 on dist-test-slave-rpg6"
server_key: "b0b8871ac27d10af4d6ea5b0b3967b62"
server_key_iv: "70468db38f7a25135f54be89c02c2ab6"
server_key_version: "encryptionkey@0"
I20260620 02:35:23.085304 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "277fa73821eb4e3384d1f8bfb09ea599"
format_stamp: "Formatted at 2026-06-20 02:35:23 on dist-test-slave-rpg6"
server_key: "b0b8871ac27d10af4d6ea5b0b3967b62"
server_key_iv: "70468db38f7a25135f54be89c02c2ab6"
server_key_version: "encryptionkey@0"
I20260620 02:35:23.091421 1160 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:35:23.096668 1175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:23.098706 1160 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260620 02:35:23.098977 1160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "277fa73821eb4e3384d1f8bfb09ea599"
format_stamp: "Formatted at 2026-06-20 02:35:23 on dist-test-slave-rpg6"
server_key: "b0b8871ac27d10af4d6ea5b0b3967b62"
server_key_iv: "70468db38f7a25135f54be89c02c2ab6"
server_key_version: "encryptionkey@0"
I20260620 02:35:23.099233 1160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:23.127960 1160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:23.133158 1160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:23.133409 1160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:23.134308 1160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:23.135653 1160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:23.135726 1160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:23.135810 1160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:23.135921 1160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 02:35:23.172753 1160 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46093
I20260620 02:35:23.172848 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46093 every 8 connection(s)
I20260620 02:35:23.174219 1160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:35:23.179212 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 1160
I20260620 02:35:23.179451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:35:23.180053 388 external_mini_cluster.cc:1428] Setting key 9a92ad30e8573a8567448f9a99bc5148
Jun 20 02:35:23 dist-test-slave-rpg6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:35:23.204038 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:23.177322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45555 (local address 127.0.97.62:39233)
0620 02:35:23.177984 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:23.177993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:23.178789 (+ 796us) server_negotiation.cc:407] Connection header received
0620 02:35:23.181634 (+ 2845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:23.181642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:23.181745 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:23.181949 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:23.184312 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.185233 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.188414 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.189205 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.189387 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:23.195015 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:23.195028 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:23.195034 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:23.195092 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:23.199045 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:23.200998 (+ 1953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:23.201003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:23.201006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:23.201169 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:23.201754 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:23.201762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:23.201766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:23.202120 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:23.202247 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:23.203202 (+ 955us) server_negotiation.cc:299] Negotiation successful
0620 02:35:23.203584 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":327,"threads_started":1}
I20260620 02:35:23.205358 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:23.177656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39233 (local address 127.0.97.2:45555)
0620 02:35:23.178554 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:23.178598 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:23.181230 (+ 2632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:23.182212 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:23.182222 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:23.183103 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:23.184066 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:23.184097 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.185430 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.185438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:23.188187 (+ 2749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:23.188214 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.189686 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.189694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:23.190089 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:23.191111 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:23.191146 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:23.194517 (+ 3371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:23.199399 (+ 4882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:23.199413 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:23.199439 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:23.200556 (+ 1117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:23.201295 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:23.201300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:23.201303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:23.201527 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:23.202259 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:23.202266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:23.202653 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 02:35:23.202997 (+ 344us) client_negotiation.cc:240] Negotiation successful
0620 02:35:23.203967 (+ 970us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":334,"threads_started":1}
I20260620 02:35:23.207564 1288 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39233
I20260620 02:35:23.208031 1288 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:23.209390 1288 heartbeater.cc:511] Master 127.0.97.62:39233 requested a full tablet report, sending...
I20260620 02:35:23.211628 965 ts_manager.cc:194] Registered new tserver with Master: 277fa73821eb4e3384d1f8bfb09ea599 (127.0.97.2:46093)
I20260620 02:35:23.213070 965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:35:23.234005 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39233
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:23.604559 1294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:23.604931 1294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:23.605300 1294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:23.613919 1294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:23.614092 1294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:23.614370 1294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:35:23.625741 1294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39233
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:23.627226 1294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:23.628775 1294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 02:35:23.636646 1154 heartbeater.cc:503] Master 127.0.97.62:39233 was elected leader, sending a full tablet report...
W20260620 02:35:23.645401 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:23.645334 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:23.645422 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:23.646243 1294 server_base.cc:1034] running on GCE node
I20260620 02:35:23.646966 1294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:23.648154 1294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:23.649398 1294 hybrid_clock.cc:648] HybridClock initialized: now 1781922923649322 us; error 68 us; skew 500 ppm
I20260620 02:35:23.649834 1294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:23 dist-test-slave-rpg6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:23.653899 1294 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:35:23.655887 1294 webserver.cc:460] Webserver started at http://127.0.97.3:40999/ using document root <none> and password file <none>
I20260620 02:35:23.656690 1294 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:23.656855 1294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:23.657187 1294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:23.660468 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "001cee2c01af4f35ad3d0b0407468de3"
format_stamp: "Formatted at 2026-06-20 02:35:23 on dist-test-slave-rpg6"
server_key: "0ba80b87d56cb337c4958a9126c3b1b7"
server_key_iv: "9dea88dd534490f86f6d048e14c8ebdf"
server_key_version: "encryptionkey@0"
I20260620 02:35:23.661381 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "001cee2c01af4f35ad3d0b0407468de3"
format_stamp: "Formatted at 2026-06-20 02:35:23 on dist-test-slave-rpg6"
server_key: "0ba80b87d56cb337c4958a9126c3b1b7"
server_key_iv: "9dea88dd534490f86f6d048e14c8ebdf"
server_key_version: "encryptionkey@0"
I20260620 02:35:23.668073 1294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:35:23.672309 1309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:23.673934 1294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:35:23.674177 1294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "001cee2c01af4f35ad3d0b0407468de3"
format_stamp: "Formatted at 2026-06-20 02:35:23 on dist-test-slave-rpg6"
server_key: "0ba80b87d56cb337c4958a9126c3b1b7"
server_key_iv: "9dea88dd534490f86f6d048e14c8ebdf"
server_key_version: "encryptionkey@0"
I20260620 02:35:23.674531 1294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:23.700671 1294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:23.706400 1294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:23.706772 1294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:23.707883 1294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:23.709960 1294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:23.710080 1294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:23.710237 1294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:23.710314 1294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:23.751025 1294 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39315
I20260620 02:35:23.751042 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39315 every 8 connection(s)
I20260620 02:35:23.752676 1294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:35:23.756073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 1294
I20260620 02:35:23.756251 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:35:23.756664 388 external_mini_cluster.cc:1428] Setting key 218221adff46991deebfa0bb0ce99b9d
Jun 20 02:35:23 dist-test-slave-rpg6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:23.779959 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:23.757819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34717 (local address 127.0.97.62:39233)
0620 02:35:23.758541 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:23.758547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:23.758564 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:35:23.761437 (+ 2873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:23.761443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:23.761580 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:23.761718 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:23.764809 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.765602 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.767420 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.767974 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.768119 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:23.773326 (+ 5207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:23.773335 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:23.773340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:23.773386 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:23.776571 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:23.777471 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:23.777475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:23.777477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:23.777602 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:23.778056 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:23.778065 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:23.778067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:23.778363 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:23.778518 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:23.779274 (+ 756us) server_negotiation.cc:299] Negotiation successful
0620 02:35:23.779533 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":745,"threads_started":1}
I20260620 02:35:23.781067 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:23.757090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39233 (local address 127.0.97.3:34717)
0620 02:35:23.758174 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:23.758241 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:23.760785 (+ 2544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:23.762152 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:23.762166 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:23.763359 (+ 1193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:23.764326 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:23.764355 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.765817 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.765828 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:23.767217 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:23.767243 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.768256 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.768264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:23.768520 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:23.769334 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:23.769360 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:23.773028 (+ 3668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:23.776759 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:23.776771 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:23.776793 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:23.777296 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:23.777714 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:23.777722 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:23.777726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:23.777931 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:23.778494 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:23.778503 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:23.778889 (+ 386us) client_negotiation.cc:769] Sending connection context
0620 02:35:23.779171 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 02:35:23.779772 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":446,"threads_started":1}
I20260620 02:35:23.782846 1422 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39233
I20260620 02:35:23.783200 1422 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:23.784077 1422 heartbeater.cc:511] Master 127.0.97.62:39233 requested a full tablet report, sending...
I20260620 02:35:23.786113 964 ts_manager.cc:194] Registered new tserver with Master: 001cee2c01af4f35ad3d0b0407468de3 (127.0.97.3:39315)
I20260620 02:35:23.787278 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34717
I20260620 02:35:23.789250 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:35:23.812786 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:23.797348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.0.97.62:39233)
0620 02:35:23.797670 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:23.797678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:23.798970 (+ 1292us) server_negotiation.cc:407] Connection header received
0620 02:35:23.799097 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:23.799103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:23.799203 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:23.799356 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:23.801077 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.801957 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.803402 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.803911 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.804026 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:23.807065 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:23.807070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:23.807074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:23.807112 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:23.809712 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:23.810612 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:23.810617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:23.810619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:23.810759 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:23.811212 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:23.811216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:23.811218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:23.811599 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:23.811788 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:23.812333 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 02:35:23.812543 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:35:23.820230 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:35:23.827709 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:35:23.868140 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:23.842990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46093 (local address 127.0.0.1:38932)
0620 02:35:23.855144 (+ 12154us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:23.856133 (+ 989us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:23.856418 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:23.858034 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:23.858046 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:23.858605 (+ 559us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:23.859221 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:23.859235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.861555 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.861560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:23.864888 (+ 3328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:23.864903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.866388 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.866395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:23.867087 (+ 692us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:23.867153 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 02:35:23.867286 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 02:35:23.867514 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11750,"mutex_wait_us":321,"thread_start_us":14277,"threads_started":1}
I20260620 02:35:23.868177 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:23.846906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40479 (local address 127.0.0.1:40882)
0620 02:35:23.859071 (+ 12165us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:23.859089 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:23.859233 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:23.860031 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:23.860050 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:23.860077 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:23.860425 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:23.860436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.862375 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.862380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:23.865226 (+ 2846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:23.865238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.867135 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.867142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:23.867341 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:23.867374 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:35:23.867501 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 02:35:23.867633 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11992,"mutex_wait_us":231,"thread_start_us":11808,"threads_started":1}
I20260620 02:35:23.869059 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:23.848203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.0.97.1:40479)
0620 02:35:23.857998 (+ 9795us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:23.858007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:23.859393 (+ 1386us) server_negotiation.cc:407] Connection header received
0620 02:35:23.859506 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:23.859514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:23.859784 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:23.860026 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:23.860619 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.862191 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.865411 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.866906 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.867060 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:23.867297 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:23.868507 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0620 02:35:23.868822 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9622,"thread_start_us":9434,"threads_started":1}
I20260620 02:35:23.869258 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:23.843403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.0.97.2:46093)
0620 02:35:23.857229 (+ 13826us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:23.857243 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:23.857281 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:35:23.857404 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:23.857415 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:23.857750 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:23.859453 (+ 1703us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:23.859703 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.861400 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.865116 (+ 3716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.866182 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.866357 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:23.866569 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:23.868201 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0620 02:35:23.868959 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13509,"thread_start_us":13366,"threads_started":1}
I20260620 02:35:23.870249 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:23.843169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39315 (local address 127.0.0.1:46818)
0620 02:35:23.857584 (+ 14415us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:23.857610 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:23.857773 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:23.858830 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:23.858835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:23.858872 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:23.859384 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:23.859395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.861384 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.861389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:23.864606 (+ 3217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:23.864621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.869448 (+ 4827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.869460 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:23.869602 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:23.869628 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 02:35:23.869746 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 02:35:23.869881 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14269,"mutex_wait_us":212,"thread_start_us":12038,"threads_started":1}
I20260620 02:35:23.870355 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:23.843805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.0.97.3:39315)
0620 02:35:23.844529 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:23.844543 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:23.857921 (+ 13378us) server_negotiation.cc:407] Connection header received
0620 02:35:23.858038 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:23.858049 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:23.858303 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:23.858490 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:23.859597 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.861214 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.864796 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:23.866172 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:23.866354 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:23.866927 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:23.869802 (+ 2875us) server_negotiation.cc:299] Negotiation successful
0620 02:35:23.869992 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":336,"threads_started":1}
I20260620 02:35:23.873426 1089 tablet_service.cc:1467] Processing CreateTablet for tablet 711be25565fb4fce9a67b8c2a5590e5b (DEFAULT_TABLE table=test-table [id=eb51cafc32c946d2b262fc848850ae87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:23.873976 1223 tablet_service.cc:1467] Processing CreateTablet for tablet 711be25565fb4fce9a67b8c2a5590e5b (DEFAULT_TABLE table=test-table [id=eb51cafc32c946d2b262fc848850ae87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:23.873863 1357 tablet_service.cc:1467] Processing CreateTablet for tablet 711be25565fb4fce9a67b8c2a5590e5b (DEFAULT_TABLE table=test-table [id=eb51cafc32c946d2b262fc848850ae87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:23.875046 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711be25565fb4fce9a67b8c2a5590e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:23.875396 1089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711be25565fb4fce9a67b8c2a5590e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:23.875396 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711be25565fb4fce9a67b8c2a5590e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:23.888316 1441 tablet_bootstrap.cc:492] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599: Bootstrap starting.
I20260620 02:35:23.892865 1442 tablet_bootstrap.cc:492] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3: Bootstrap starting.
I20260620 02:35:23.893726 1441 tablet_bootstrap.cc:654] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:23.895244 1441 log.cc:824] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:23.897018 1443 tablet_bootstrap.cc:492] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539: Bootstrap starting.
I20260620 02:35:23.897075 1442 tablet_bootstrap.cc:654] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:23.898844 1442 log.cc:824] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:23.899410 1441 tablet_bootstrap.cc:492] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599: No bootstrap required, opened a new log
I20260620 02:35:23.899740 1441 ts_tablet_manager.cc:1397] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260620 02:35:23.901806 1443 tablet_bootstrap.cc:654] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:23.903576 1443 log.cc:824] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:23.912429 1441 raft_consensus.cc:348] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277fa73821eb4e3384d1f8bfb09ea599" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46093 } } peers { permanent_uuid: "001cee2c01af4f35ad3d0b0407468de3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } } peers { permanent_uuid: "6398e80f43a54163bc2ed589b126a539" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40479 } }
I20260620 02:35:23.913296 1441 raft_consensus.cc:374] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:23.913445 1441 raft_consensus.cc:729] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277fa73821eb4e3384d1f8bfb09ea599, State: Initialized, Role: FOLLOWER
I20260620 02:35:23.915433 1441 consensus_queue.cc:260] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277fa73821eb4e3384d1f8bfb09ea599" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46093 } } peers { permanent_uuid: "001cee2c01af4f35ad3d0b0407468de3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } } peers { permanent_uuid: "6398e80f43a54163bc2ed589b126a539" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40479 } }
I20260620 02:35:23.917946 1443 tablet_bootstrap.cc:492] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539: No bootstrap required, opened a new log
I20260620 02:35:23.918414 1443 ts_tablet_manager.cc:1397] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20260620 02:35:23.923508 1288 heartbeater.cc:503] Master 127.0.97.62:39233 was elected leader, sending a full tablet report...
I20260620 02:35:23.924784 1441 ts_tablet_manager.cc:1428] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599: Time spent starting tablet: real 0.025s user 0.013s sys 0.009s
W20260620 02:35:23.926353 1289 tablet.cc:2369] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:35:23.929672 1442 tablet_bootstrap.cc:492] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3: No bootstrap required, opened a new log
I20260620 02:35:23.930023 1442 ts_tablet_manager.cc:1397] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.021s
I20260620 02:35:23.933876 1443 raft_consensus.cc:348] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277fa73821eb4e3384d1f8bfb09ea599" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46093 } } peers { permanent_uuid: "001cee2c01af4f35ad3d0b0407468de3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } } peers { permanent_uuid: "6398e80f43a54163bc2ed589b126a539" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40479 } }
I20260620 02:35:23.934409 1443 raft_consensus.cc:374] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:23.934512 1443 raft_consensus.cc:729] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398e80f43a54163bc2ed589b126a539, State: Initialized, Role: FOLLOWER
I20260620 02:35:23.935653 1443 consensus_queue.cc:260] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277fa73821eb4e3384d1f8bfb09ea599" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46093 } } peers { permanent_uuid: "001cee2c01af4f35ad3d0b0407468de3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } } peers { permanent_uuid: "6398e80f43a54163bc2ed589b126a539" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40479 } }
I20260620 02:35:23.938936 1443 ts_tablet_manager.cc:1428] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260620 02:35:23.941844 1442 raft_consensus.cc:348] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277fa73821eb4e3384d1f8bfb09ea599" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46093 } } peers { permanent_uuid: "001cee2c01af4f35ad3d0b0407468de3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } } peers { permanent_uuid: "6398e80f43a54163bc2ed589b126a539" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40479 } }
I20260620 02:35:23.942358 1442 raft_consensus.cc:374] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:23.942481 1442 raft_consensus.cc:729] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001cee2c01af4f35ad3d0b0407468de3, State: Initialized, Role: FOLLOWER
I20260620 02:35:23.943154 1442 consensus_queue.cc:260] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277fa73821eb4e3384d1f8bfb09ea599" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46093 } } peers { permanent_uuid: "001cee2c01af4f35ad3d0b0407468de3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } } peers { permanent_uuid: "6398e80f43a54163bc2ed589b126a539" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40479 } }
I20260620 02:35:23.944402 1422 heartbeater.cc:503] Master 127.0.97.62:39233 was elected leader, sending a full tablet report...
I20260620 02:35:23.945128 1442 ts_tablet_manager.cc:1428] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260620 02:35:24.004961 1423 tablet.cc:2369] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:35:24.101346 1155 tablet.cc:2369] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:35:24.185249 1447 raft_consensus.cc:482] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:35:24.185624 1447 raft_consensus.cc:504] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277fa73821eb4e3384d1f8bfb09ea599" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46093 } } peers { permanent_uuid: "001cee2c01af4f35ad3d0b0407468de3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } } peers { permanent_uuid: "6398e80f43a54163bc2ed589b126a539" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40479 } }
I20260620 02:35:24.187568 1447 leader_election.cc:288] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 001cee2c01af4f35ad3d0b0407468de3 (127.0.97.3:39315), 6398e80f43a54163bc2ed589b126a539 (127.0.97.1:40479)
I20260620 02:35:24.196018 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:24.188043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45653 (local address 127.0.97.3:39315)
0620 02:35:24.188361 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:24.188370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:24.189717 (+ 1347us) server_negotiation.cc:407] Connection header received
0620 02:35:24.189915 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:24.189921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:24.190002 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:24.190185 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:24.190943 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.191820 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.193887 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.194877 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.194982 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:24.195062 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:24.195580 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 02:35:24.195742 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260620 02:35:24.196034 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:24.188038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39315 (local address 127.0.97.2:45653)
0620 02:35:24.189507 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:24.189525 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:24.189683 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:24.190329 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:24.190333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:24.190356 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:24.190788 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:24.190801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.192015 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.192042 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:24.193718 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:24.193730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.195113 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.195119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:24.195312 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:24.195349 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:35:24.195466 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 02:35:24.195623 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"thread_start_us":569,"threads_started":1}
I20260620 02:35:24.196578 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:24.188082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54063 (local address 127.0.97.1:40479)
0620 02:35:24.188362 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:24.188370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:24.189956 (+ 1586us) server_negotiation.cc:407] Connection header received
0620 02:35:24.190306 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:24.190311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:24.190405 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:24.190510 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:24.191361 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.192302 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.194451 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.195480 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.195607 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:24.195725 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:24.196167 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 02:35:24.196315 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:35:24.196620 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:24.188207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40479 (local address 127.0.97.2:54063)
0620 02:35:24.189715 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:24.189737 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:24.189976 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:24.190559 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:24.190566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:24.190627 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:24.191028 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:24.191043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.192556 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.192560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:24.194229 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:24.194243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.195725 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.195741 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:24.195972 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:24.196007 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:35:24.196117 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 02:35:24.196312 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1241,"thread_start_us":404,"threads_started":1}
I20260620 02:35:24.197455 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711be25565fb4fce9a67b8c2a5590e5b" candidate_uuid: "277fa73821eb4e3384d1f8bfb09ea599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001cee2c01af4f35ad3d0b0407468de3" is_pre_election: true
I20260620 02:35:24.197618 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711be25565fb4fce9a67b8c2a5590e5b" candidate_uuid: "277fa73821eb4e3384d1f8bfb09ea599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398e80f43a54163bc2ed589b126a539" is_pre_election: true
I20260620 02:35:24.197963 1377 raft_consensus.cc:2454] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277fa73821eb4e3384d1f8bfb09ea599 in term 0.
I20260620 02:35:24.198175 1109 raft_consensus.cc:2454] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277fa73821eb4e3384d1f8bfb09ea599 in term 0.
I20260620 02:35:24.198757 1176 leader_election.cc:302] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001cee2c01af4f35ad3d0b0407468de3, 277fa73821eb4e3384d1f8bfb09ea599; no voters:
I20260620 02:35:24.199296 1447 raft_consensus.cc:2781] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:35:24.199466 1447 raft_consensus.cc:482] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:35:24.199601 1447 raft_consensus.cc:3037] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:24.202312 1447 raft_consensus.cc:504] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277fa73821eb4e3384d1f8bfb09ea599" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46093 } } peers { permanent_uuid: "001cee2c01af4f35ad3d0b0407468de3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } } peers { permanent_uuid: "6398e80f43a54163bc2ed589b126a539" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40479 } }
I20260620 02:35:24.203121 1447 leader_election.cc:288] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [CANDIDATE]: Term 1 election: Requested vote from peers 001cee2c01af4f35ad3d0b0407468de3 (127.0.97.3:39315), 6398e80f43a54163bc2ed589b126a539 (127.0.97.1:40479)
I20260620 02:35:24.204020 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711be25565fb4fce9a67b8c2a5590e5b" candidate_uuid: "277fa73821eb4e3384d1f8bfb09ea599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001cee2c01af4f35ad3d0b0407468de3"
I20260620 02:35:24.204404 1377 raft_consensus.cc:3037] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:24.204429 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711be25565fb4fce9a67b8c2a5590e5b" candidate_uuid: "277fa73821eb4e3384d1f8bfb09ea599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6398e80f43a54163bc2ed589b126a539"
I20260620 02:35:24.204689 1109 raft_consensus.cc:3037] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:24.207041 1377 raft_consensus.cc:2454] T 711be25565fb4fce9a67b8c2a5590e5b P 001cee2c01af4f35ad3d0b0407468de3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277fa73821eb4e3384d1f8bfb09ea599 in term 1.
I20260620 02:35:24.207549 1109 raft_consensus.cc:2454] T 711be25565fb4fce9a67b8c2a5590e5b P 6398e80f43a54163bc2ed589b126a539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277fa73821eb4e3384d1f8bfb09ea599 in term 1.
I20260620 02:35:24.207716 1176 leader_election.cc:302] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001cee2c01af4f35ad3d0b0407468de3, 277fa73821eb4e3384d1f8bfb09ea599; no voters:
I20260620 02:35:24.208141 1447 raft_consensus.cc:2781] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:24.208714 1447 raft_consensus.cc:686] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [term 1 LEADER]: Becoming Leader. State: Replica: 277fa73821eb4e3384d1f8bfb09ea599, State: Running, Role: LEADER
I20260620 02:35:24.209239 1447 consensus_queue.cc:237] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277fa73821eb4e3384d1f8bfb09ea599" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46093 } } peers { permanent_uuid: "001cee2c01af4f35ad3d0b0407468de3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } } peers { permanent_uuid: "6398e80f43a54163bc2ed589b126a539" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40479 } }
I20260620 02:35:24.214617 965 catalog_manager.cc:5495] T 711be25565fb4fce9a67b8c2a5590e5b P 277fa73821eb4e3384d1f8bfb09ea599 reported cstate change: term changed from 0 to 1, leader changed from <none> to 277fa73821eb4e3384d1f8bfb09ea599 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "277fa73821eb4e3384d1f8bfb09ea599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277fa73821eb4e3384d1f8bfb09ea599" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001cee2c01af4f35ad3d0b0407468de3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6398e80f43a54163bc2ed589b126a539" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40479 } health_report { overall_health: UNKNOWN } } }
Jun 20 02:35:24 dist-test-slave-rpg6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:35:24.260869 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:24.242159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.0.97.1:40479)
0620 02:35:24.242438 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:24.242448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:24.242929 (+ 481us) server_negotiation.cc:407] Connection header received
0620 02:35:24.243163 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:24.243168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:24.243266 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:24.243380 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:24.244977 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.245751 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.247116 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.247568 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.247766 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:24.253201 (+ 5435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:24.253219 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:24.253236 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:24.253307 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:24.256796 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.257582 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.257589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.257592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.257756 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.258630 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.258635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.258637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.259073 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:24.259337 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:24.260236 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 02:35:24.260503 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 02:35:24 dist-test-slave-rpg6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:35:24 dist-test-slave-rpg6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:35:24.315760 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:24.299536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.0.97.1:40479)
0620 02:35:24.300072 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:24.300077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:24.300150 (+ 73us) server_negotiation.cc:407] Connection header received
0620 02:35:24.300398 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:24.300402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:24.300468 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:24.300554 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:24.301838 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.302570 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.304150 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.304693 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.304844 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:24.309801 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:24.309807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:24.309811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:24.309854 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:24.312868 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.313855 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.313860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.313862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.314010 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.314347 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.314350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.314352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.314610 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:24.314792 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:24.315120 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 02:35:24.315328 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408}
Jun 20 02:35:24 dist-test-slave-rpg6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:35:24 dist-test-slave-rpg6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:35:24.365583 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:24.349871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.0.97.1:40479)
0620 02:35:24.350155 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:24.350162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:24.350474 (+ 312us) server_negotiation.cc:407] Connection header received
0620 02:35:24.350583 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:24.350588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:24.350688 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:24.350843 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:24.352413 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.353353 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.354752 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.355525 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.355653 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:24.360147 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:24.360153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:24.360156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:24.360200 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:24.362825 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.363654 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.363659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.363661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.363810 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.364209 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.364213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.364215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.364524 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:24.364715 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:24.365048 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 02:35:24.365316 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 02:35:24.367785 1069 tablet_service.cc:2908] Rejecting scan request for tablet 711be25565fb4fce9a67b8c2a5590e5b: Uninitialized: clean time has not yet been initialized
I20260620 02:35:24.371134 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 1026
I20260620 02:35:24.386407 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:24.372048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.0.97.1:40479)
0620 02:35:24.372254 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:24.372259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:24.372648 (+ 389us) server_negotiation.cc:407] Connection header received
0620 02:35:24.372989 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:24.372993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:24.373075 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:24.373179 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:24.374438 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.375119 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.376554 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.376969 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.377120 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:24.380531 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:24.380537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:24.380541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:24.380591 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:24.383194 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.383999 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.384009 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.384012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.384134 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.384502 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.384507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.384510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.384814 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:24.385109 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:24.385849 (+ 740us) server_negotiation.cc:299] Negotiation successful
0620 02:35:24.386072 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 02:35:24.387082 1149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:24.480285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 1026
I20260620 02:35:24.497434 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 1160
Jun 20 02:35:24 dist-test-slave-rpg6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:35:24.514290 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:24.498709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.0.97.2:46093)
0620 02:35:24.499229 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:24.499233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:24.499251 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:35:24.499303 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:24.499307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:24.499363 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:24.499459 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:24.500762 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.501483 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.502760 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.503286 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.503391 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:24.508169 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:24.508185 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:24.508202 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:24.508251 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:24.511217 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.512114 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.512121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.512124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.512285 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.512682 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.512687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.512689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.513069 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:24.513360 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:24.513691 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 02:35:24.513892 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":316,"threads_started":1}
I20260620 02:35:24.514959 1283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:24.607759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 1160
I20260620 02:35:24.625779 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 1294
Jun 20 02:35:24 dist-test-slave-rpg6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:35:24.642409 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:24.626624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.0.97.3:39315)
0620 02:35:24.626818 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:24.626823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:24.626841 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:35:24.626988 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:24.626992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:24.627062 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:24.627155 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:24.628664 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.629375 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.630648 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:24.631078 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:24.631250 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:24.635729 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:24.635746 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:24.635772 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:24.635813 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:24.639130 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.640147 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.640156 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.640159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.640316 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:24.640699 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:24.640703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:24.640705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:24.641181 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:24.641408 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:24.641753 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 02:35:24.641943 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:35:24.643196 1417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:24.732764 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 1294
I20260620 02:35:24.749915 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 933
I20260620 02:35:24.751291 995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:24.860942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 933
2026-06-20T02:35:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4727 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:35:24 dist-test-slave-rpg6 krb5kdc[1472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:35:24 dist-test-slave-rpg6 krb5kdc[1472](info): set up 2 sockets
Jun 20 02:35:24 dist-test-slave-rpg6 krb5kdc[1472](info): commencing operation
krb5kdc: starting...
W20260620 02:35:26.000286 388 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:35:26 dist-test-slave-rpg6 krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:35:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:35:26.217664 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41785
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:26.593135 1485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:26.593466 1485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:26.593631 1485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:26.601521 1485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:35:26.601655 1485 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:35:26.601742 1485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:26.601840 1485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:26.601907 1485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:35:26.601994 1485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:35:26.612639 1485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41785
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:26.614081 1485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:26.615579 1485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:26.629647 1490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:26.629731 1491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:26.629917 1485 server_base.cc:1034] running on GCE node
W20260620 02:35:26.629752 1493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:26.631145 1485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:26.632987 1485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:26.634369 1485 hybrid_clock.cc:648] HybridClock initialized: now 1781922926634259 us; error 40 us; skew 500 ppm
I20260620 02:35:26.635089 1485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:26 dist-test-slave-rpg6 krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:26.638978 1485 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:35:26.640657 1485 webserver.cc:460] Webserver started at http://127.0.97.62:37335/ using document root <none> and password file <none>
I20260620 02:35:26.641408 1485 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:26.641502 1485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:26.641776 1485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:26.644982 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "cbf937f8a990411c95c5b920e8ec0b3c"
format_stamp: "Formatted at 2026-06-20 02:35:26 on dist-test-slave-rpg6"
server_key: "14938eecad66bfd9a7e454a21dd1804c"
server_key_iv: "251f38f02e6fe509d134db68f79a9b39"
server_key_version: "encryptionkey@0"
I20260620 02:35:26.645820 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "cbf937f8a990411c95c5b920e8ec0b3c"
format_stamp: "Formatted at 2026-06-20 02:35:26 on dist-test-slave-rpg6"
server_key: "14938eecad66bfd9a7e454a21dd1804c"
server_key_iv: "251f38f02e6fe509d134db68f79a9b39"
server_key_version: "encryptionkey@0"
I20260620 02:35:26.651352 1485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:35:26.655241 1500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:26.657116 1485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:35:26.657343 1485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "cbf937f8a990411c95c5b920e8ec0b3c"
format_stamp: "Formatted at 2026-06-20 02:35:26 on dist-test-slave-rpg6"
server_key: "14938eecad66bfd9a7e454a21dd1804c"
server_key_iv: "251f38f02e6fe509d134db68f79a9b39"
server_key_version: "encryptionkey@0"
I20260620 02:35:26.657600 1485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:26.674649 1485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:26.680312 1485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:26.680576 1485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:26.705955 1485 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41785
I20260620 02:35:26.705942 1561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41785 every 8 connection(s)
I20260620 02:35:26.707666 1485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:35:26.709069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 1485
I20260620 02:35:26.709391 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:35:26.710078 388 external_mini_cluster.cc:1428] Setting key 3eb9a4c6874c95f38dce7e8837fbaa66
I20260620 02:35:26.712734 1562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:35:26 dist-test-slave-rpg6 krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:26.727579 1562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c: Bootstrap starting.
I20260620 02:35:26.732352 1562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:26.733909 1562 log.cc:824] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:26.737298 1562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c: No bootstrap required, opened a new log
I20260620 02:35:26.738247 1565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:26.712580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.0.97.62:41785)
0620 02:35:26.713537 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:26.713553 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:26.713604 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:35:26.716019 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:26.716053 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:26.716689 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:26.717228 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:26.718963 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:26.721131 (+ 2168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:26.722613 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:26.723749 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:26.723981 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:26.728312 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:26.728324 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:26.728346 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:26.728399 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:26.733540 (+ 5141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:26.734526 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:26.734534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:26.734541 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:26.734860 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:26.735275 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:26.735281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:26.735284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:26.735892 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:26.736229 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:26.736656 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 02:35:26.737144 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":443,"threads_started":1}
I20260620 02:35:26.746690 1562 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf937f8a990411c95c5b920e8ec0b3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260620 02:35:26.747120 1562 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:26.747264 1562 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf937f8a990411c95c5b920e8ec0b3c, State: Initialized, Role: FOLLOWER
I20260620 02:35:26.748015 1562 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf937f8a990411c95c5b920e8ec0b3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260620 02:35:26.748334 1562 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:35:26.748488 1562 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:35:26.748705 1562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:26.751084 1562 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf937f8a990411c95c5b920e8ec0b3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260620 02:35:26.751631 1562 leader_election.cc:302] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf937f8a990411c95c5b920e8ec0b3c; no voters:
I20260620 02:35:26.752494 1562 leader_election.cc:288] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:35:26.752660 1567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:26.753480 1567 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [term 1 LEADER]: Becoming Leader. State: Replica: cbf937f8a990411c95c5b920e8ec0b3c, State: Running, Role: LEADER
I20260620 02:35:26.754051 1567 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf937f8a990411c95c5b920e8ec0b3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } }
I20260620 02:35:26.755115 1562 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:35:26.757732 1569 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf937f8a990411c95c5b920e8ec0b3c. Latest consensus state: current_term: 1 leader_uuid: "cbf937f8a990411c95c5b920e8ec0b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf937f8a990411c95c5b920e8ec0b3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } } }
I20260620 02:35:26.757699 1568 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf937f8a990411c95c5b920e8ec0b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf937f8a990411c95c5b920e8ec0b3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41785 } } }
I20260620 02:35:26.758424 1569 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:26.758455 1568 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:26.760514 1575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:35:26.766885 1575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:35:26.777601 1575 catalog_manager.cc:1347] Generated new cluster ID: 5859323ba11a4f6ab74167df911fadbf
I20260620 02:35:26.777751 1575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:35:26.795331 1575 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:35:26.796895 1575 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:35:26.811033 1575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c: Generated new TSK 0
I20260620 02:35:26.812292 1575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:35:26.866143 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41785
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:27.233980 1588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:27.234328 1588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:27.234447 1588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:27.244978 1588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:27.245170 1588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:27.245267 1588 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:35:27.245361 1588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:35:27.256705 1588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:27.258095 1588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:27.259639 1588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:27.274931 1594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:27.275128 1596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:27.275557 1593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:27.276060 1588 server_base.cc:1034] running on GCE node
I20260620 02:35:27.276885 1588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:27.278146 1588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:27.279614 1588 hybrid_clock.cc:648] HybridClock initialized: now 1781922927279406 us; error 188 us; skew 500 ppm
I20260620 02:35:27.280030 1588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:27 dist-test-slave-rpg6 krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:27.288559 1588 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:35:27.290448 1588 webserver.cc:460] Webserver started at http://127.0.97.1:39289/ using document root <none> and password file <none>
I20260620 02:35:27.291234 1588 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:27.291332 1588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:27.291625 1588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:27.294896 1588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "95d614b3bbd54d98b3d6fda0678383a0"
format_stamp: "Formatted at 2026-06-20 02:35:27 on dist-test-slave-rpg6"
server_key: "445e11b0a58cf84e98199145cbfb162c"
server_key_iv: "6df6dc037ba4fde68c4b403d8d579709"
server_key_version: "encryptionkey@0"
I20260620 02:35:27.295797 1588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "95d614b3bbd54d98b3d6fda0678383a0"
format_stamp: "Formatted at 2026-06-20 02:35:27 on dist-test-slave-rpg6"
server_key: "445e11b0a58cf84e98199145cbfb162c"
server_key_iv: "6df6dc037ba4fde68c4b403d8d579709"
server_key_version: "encryptionkey@0"
I20260620 02:35:27.301759 1588 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:35:27.305775 1603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:27.307502 1588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:35:27.307713 1588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "95d614b3bbd54d98b3d6fda0678383a0"
format_stamp: "Formatted at 2026-06-20 02:35:27 on dist-test-slave-rpg6"
server_key: "445e11b0a58cf84e98199145cbfb162c"
server_key_iv: "6df6dc037ba4fde68c4b403d8d579709"
server_key_version: "encryptionkey@0"
I20260620 02:35:27.308025 1588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:27.328025 1588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:27.333979 1588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:27.334316 1588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:27.337349 1588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:27.337494 1588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:27.337625 1588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:27.337707 1588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:35:27 dist-test-slave-rpg6 krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:27.367131 1612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:27.338216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58907 (local address 127.0.97.62:41785)
0620 02:35:27.338707 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:27.338712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:27.340838 (+ 2126us) server_negotiation.cc:407] Connection header received
0620 02:35:27.343521 (+ 2683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:27.343532 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:27.343637 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:27.343727 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:27.346968 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.348093 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.350521 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.351345 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.351531 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:27.357768 (+ 6237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:27.357774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:27.357777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:27.357815 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:27.362552 (+ 4737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:27.363534 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:27.363543 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:27.363550 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:27.363703 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:27.364614 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:27.364619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:27.364621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:27.364965 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:27.365147 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:27.366481 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0620 02:35:27.366747 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":255,"threads_started":1}
I20260620 02:35:27.369316 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:27.338967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:58907)
0620 02:35:27.340610 (+ 1643us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:27.340665 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:27.343048 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:27.344464 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:27.344477 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:27.345437 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:27.346703 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:27.346741 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.348279 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.348287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:27.350309 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:27.350334 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.351710 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.351724 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:27.352126 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:27.353123 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:27.353151 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:27.357446 (+ 4295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:27.362791 (+ 5345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:27.362801 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:27.362822 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:27.363363 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:27.363886 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:27.363891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:27.363894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:27.364403 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:27.365171 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:27.365188 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:27.365877 (+ 689us) client_negotiation.cc:769] Sending connection context
0620 02:35:27.366323 (+ 446us) client_negotiation.cc:240] Negotiation successful
0620 02:35:27.367149 (+ 826us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":548,"threads_started":1}
I20260620 02:35:27.391060 1588 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42585
I20260620 02:35:27.391068 1723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42585 every 8 connection(s)
I20260620 02:35:27.392680 1588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:35:27.393955 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:27.376502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.1:54023)
0620 02:35:27.377511 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:27.377532 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:27.377723 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:27.378174 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:27.378185 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:27.378987 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:27.379620 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:27.379642 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.380779 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.380788 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:27.382989 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:27.383012 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.384465 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.384475 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:27.384740 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:27.385556 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:27.385582 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:27.387163 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:27.390653 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:27.390663 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:27.390675 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:27.391158 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:27.391627 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:27.391632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:27.391635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:27.391793 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:27.392454 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:27.392459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:27.392706 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 02:35:27.392863 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 02:35:27.393233 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":648,"threads_started":1}
I20260620 02:35:27.394301 1612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:27.376502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54023 (local address 127.0.97.62:41785)
0620 02:35:27.376709 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:27.376716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:27.377637 (+ 921us) server_negotiation.cc:407] Connection header received
0620 02:35:27.377886 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:27.377895 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:27.377994 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:27.378104 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:27.379822 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.380612 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.383333 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.384170 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.384337 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:27.387513 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:27.387519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:27.387523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:27.387562 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:27.390490 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:27.391356 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:27.391360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:27.391362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:27.391488 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:27.391972 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:27.391977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:27.391979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:27.392307 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:27.392429 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:27.393698 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0620 02:35:27.393969 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:35:27.395330 1724 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41785
I20260620 02:35:27.396457 1724 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:27.397269 1724 heartbeater.cc:511] Master 127.0.97.62:41785 requested a full tablet report, sending...
I20260620 02:35:27.397804 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 1588
I20260620 02:35:27.398052 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:35:27.398910 388 external_mini_cluster.cc:1428] Setting key 6e743b9a8fa6d264b233bb6fe1d13c06
I20260620 02:35:27.401912 1517 ts_manager.cc:194] Registered new tserver with Master: 95d614b3bbd54d98b3d6fda0678383a0 (127.0.97.1:42585)
I20260620 02:35:27.403437 1517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58907
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:35:27.444386 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41785
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:27.790092 1728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:27.790438 1728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:27.790619 1728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:27.798309 1728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:27.798452 1728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:27.798617 1728 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:35:27.798734 1728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:35:27.809556 1728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:27.811055 1728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:27.812610 1728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:27.827602 1733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:27.827526 1734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:27.828179 1736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:27.828652 1728 server_base.cc:1034] running on GCE node
I20260620 02:35:27.829466 1728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:27.830523 1728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:27.831768 1728 hybrid_clock.cc:648] HybridClock initialized: now 1781922927831697 us; error 40 us; skew 500 ppm
I20260620 02:35:27.832206 1728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:27 dist-test-slave-rpg6 krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:27.836030 1728 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:35:27.838069 1728 webserver.cc:460] Webserver started at http://127.0.97.2:44649/ using document root <none> and password file <none>
I20260620 02:35:27.838866 1728 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:27.838963 1728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:27.839326 1728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:27.842626 1728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "c6a3265323bd485f88484359bb93cebf"
format_stamp: "Formatted at 2026-06-20 02:35:27 on dist-test-slave-rpg6"
server_key: "ecbc42c90382acde93c581ee12c7a34b"
server_key_iv: "e4061f891b81b8a23aea6d67d2c52337"
server_key_version: "encryptionkey@0"
I20260620 02:35:27.843531 1728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c6a3265323bd485f88484359bb93cebf"
format_stamp: "Formatted at 2026-06-20 02:35:27 on dist-test-slave-rpg6"
server_key: "ecbc42c90382acde93c581ee12c7a34b"
server_key_iv: "e4061f891b81b8a23aea6d67d2c52337"
server_key_version: "encryptionkey@0"
I20260620 02:35:27.849779 1728 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 02:35:27.853952 1743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:27.855783 1728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:35:27.856077 1728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "c6a3265323bd485f88484359bb93cebf"
format_stamp: "Formatted at 2026-06-20 02:35:27 on dist-test-slave-rpg6"
server_key: "ecbc42c90382acde93c581ee12c7a34b"
server_key_iv: "e4061f891b81b8a23aea6d67d2c52337"
server_key_version: "encryptionkey@0"
I20260620 02:35:27.856341 1728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:27.894142 1728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:27.899070 1728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:27.899411 1728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:27.902310 1728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:27.902444 1728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:27.902596 1728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:27.902685 1728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:35:27 dist-test-slave-rpg6 krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:27.930148 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:27.903182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35039 (local address 127.0.97.62:41785)
0620 02:35:27.903683 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:27.903690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:27.905089 (+ 1399us) server_negotiation.cc:407] Connection header received
0620 02:35:27.908414 (+ 3325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:27.908421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:27.908534 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:27.908682 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:27.911776 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.912947 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.916504 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.916997 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.917122 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:27.922524 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:27.922530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:27.922533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:27.922571 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:27.926013 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:27.926974 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:27.926982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:27.926988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:27.927165 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:27.927936 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:27.927940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:27.927942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:27.928223 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:27.928360 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:27.929642 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0620 02:35:27.929847 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":276,"threads_started":1}
I20260620 02:35:27.932507 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:27.903954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:35039)
0620 02:35:27.904886 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:27.904948 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:27.907966 (+ 3018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:27.909069 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:27.909083 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:27.910156 (+ 1073us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:27.911415 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:27.911452 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.913172 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.913182 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:27.916253 (+ 3071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:27.916274 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.917266 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.917274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:27.917672 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:27.918771 (+ 1099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:27.918797 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:27.922180 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:27.926224 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:27.926242 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:27.926271 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:27.926813 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:27.927328 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:27.927337 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:27.927344 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:27.927744 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:27.928381 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:27.928394 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:27.929075 (+ 681us) client_negotiation.cc:769] Sending connection context
0620 02:35:27.929491 (+ 416us) client_negotiation.cc:240] Negotiation successful
0620 02:35:27.930256 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":365,"threads_started":1}
I20260620 02:35:27.957311 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:27.938801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53897 (local address 127.0.97.62:41785)
0620 02:35:27.938990 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:27.938999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:27.939980 (+ 981us) server_negotiation.cc:407] Connection header received
0620 02:35:27.940286 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:27.940292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:27.940387 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:27.940482 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:27.942800 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.943810 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.946050 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.946491 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.946643 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:27.949453 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:27.949464 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:27.949472 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:27.949531 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:27.953041 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:27.954435 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:27.954443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:27.954449 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:27.954630 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:27.955276 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:27.955281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:27.955283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:27.955681 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:27.955816 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:27.956860 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0620 02:35:27.957061 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:35:27.957898 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:27.938786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.2:53897)
0620 02:35:27.939801 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:27.939854 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:27.940057 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:27.940767 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:27.940778 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:27.941696 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:27.942569 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:27.942589 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.944100 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.944107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:27.945864 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:27.945881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:27.946716 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:27.946723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:27.946904 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:27.947785 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:27.947806 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:27.949153 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:27.953242 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:27.953255 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:27.953263 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:27.954260 (+ 997us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:27.954777 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:27.954786 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:27.954792 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:27.955013 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:27.955868 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:27.955876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:27.956160 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 02:35:27.956604 (+ 444us) client_negotiation.cc:240] Negotiation successful
0620 02:35:27.957217 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":553,"threads_started":1}
I20260620 02:35:27.969671 1728 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35435
I20260620 02:35:27.969705 1863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35435 every 8 connection(s)
I20260620 02:35:27.970993 1728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:35:27.973680 1864 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41785
I20260620 02:35:27.973984 1864 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:27.974757 1864 heartbeater.cc:511] Master 127.0.97.62:41785 requested a full tablet report, sending...
I20260620 02:35:27.975374 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 1728
I20260620 02:35:27.975610 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:35:27.976181 388 external_mini_cluster.cc:1428] Setting key c69668e329a886f4b9efabc438ed8961
I20260620 02:35:27.976703 1517 ts_manager.cc:194] Registered new tserver with Master: c6a3265323bd485f88484359bb93cebf (127.0.97.2:35435)
I20260620 02:35:27.977870 1517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35039
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:35:28.021420 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41785
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:28.384455 1868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:28.384773 1868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:28.384951 1868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:28.393378 1868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:28.393565 1868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:28.393714 1868 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:35:28.393851 1868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:35:28.404096 1868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:28.405499 1868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:28.406976 1868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 02:35:28.407934 1724 heartbeater.cc:503] Master 127.0.97.62:41785 was elected leader, sending a full tablet report...
W20260620 02:35:28.421705 1873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:28.421743 1876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:28.421670 1874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:28.422600 1868 server_base.cc:1034] running on GCE node
I20260620 02:35:28.423396 1868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:28.424540 1868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:28.425835 1868 hybrid_clock.cc:648] HybridClock initialized: now 1781922928425769 us; error 44 us; skew 500 ppm
I20260620 02:35:28.426244 1868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:28 dist-test-slave-rpg6 krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:28.430560 1868 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:35:28.432386 1868 webserver.cc:460] Webserver started at http://127.0.97.3:44837/ using document root <none> and password file <none>
I20260620 02:35:28.433208 1868 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:28.433352 1868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:28.433703 1868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:28.437080 1868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "12ec6ffbae974b98aac5771bb02f9cb8"
format_stamp: "Formatted at 2026-06-20 02:35:28 on dist-test-slave-rpg6"
server_key: "56a31ee0266e1935189c2a34470ff714"
server_key_iv: "cd4e0332fd2c38fcc251b9e9b38578a3"
server_key_version: "encryptionkey@0"
I20260620 02:35:28.438004 1868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "12ec6ffbae974b98aac5771bb02f9cb8"
format_stamp: "Formatted at 2026-06-20 02:35:28 on dist-test-slave-rpg6"
server_key: "56a31ee0266e1935189c2a34470ff714"
server_key_iv: "cd4e0332fd2c38fcc251b9e9b38578a3"
server_key_version: "encryptionkey@0"
I20260620 02:35:28.443948 1868 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:35:28.447713 1883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:28.449405 1868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:35:28.449627 1868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "12ec6ffbae974b98aac5771bb02f9cb8"
format_stamp: "Formatted at 2026-06-20 02:35:28 on dist-test-slave-rpg6"
server_key: "56a31ee0266e1935189c2a34470ff714"
server_key_iv: "cd4e0332fd2c38fcc251b9e9b38578a3"
server_key_version: "encryptionkey@0"
I20260620 02:35:28.449893 1868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:28.470185 1868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:28.475322 1868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:28.475633 1868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:28.478760 1868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:28.478885 1868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:28.479013 1868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:28.479086 1868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:35:28 dist-test-slave-rpg6 krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:28.507488 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.479547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53133 (local address 127.0.97.62:41785)
0620 02:35:28.480101 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:28.480110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:28.482222 (+ 2112us) server_negotiation.cc:407] Connection header received
0620 02:35:28.485805 (+ 3583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:28.485813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:28.485929 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:28.486042 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:28.489410 (+ 3368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.490435 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.494039 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.494585 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.494776 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.500301 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:28.500307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:28.500310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:28.500347 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:28.503164 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:28.504151 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:28.504157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:28.504159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:28.504303 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:28.505172 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:28.505179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:28.505184 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:28.505525 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:28.505692 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:28.506826 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0620 02:35:28.507072 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":338,"threads_started":1}
I20260620 02:35:28.509254 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.480875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:53133)
0620 02:35:28.481985 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:28.482046 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:28.485230 (+ 3184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:28.486451 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:28.486473 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:28.487743 (+ 1270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:28.489083 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.489127 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.490616 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.490626 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.493751 (+ 3125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.493782 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.494941 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.494952 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.495400 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.496503 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:28.496531 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:28.499976 (+ 3445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:28.503370 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:28.503382 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:28.503402 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:28.503995 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:28.504497 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:28.504506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:28.504510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:28.504984 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:28.505677 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:28.505687 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:28.506284 (+ 597us) client_negotiation.cc:769] Sending connection context
0620 02:35:28.506702 (+ 418us) client_negotiation.cc:240] Negotiation successful
0620 02:35:28.507337 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":414,"threads_started":1}
I20260620 02:35:28.531328 1868 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38691
I20260620 02:35:28.531349 2003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38691 every 8 connection(s)
I20260620 02:35:28.532847 1868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:35:28.533762 1964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.516097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.97.3:35467)
0620 02:35:28.517084 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:28.517101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:28.517405 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:28.518011 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:28.518022 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:28.518801 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:28.519284 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.519305 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.521073 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.521081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.522497 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.522513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.523622 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.523631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.523811 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.524726 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:28.524762 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:28.526415 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:28.530798 (+ 4383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:28.530807 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:28.530812 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:28.531296 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:28.531791 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:28.531797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:28.531801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:28.532005 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:28.532647 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:28.532652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:28.532849 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 02:35:28.532995 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 02:35:28.533287 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":508,"threads_started":1}
I20260620 02:35:28.534476 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.516061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35467 (local address 127.0.97.62:41785)
0620 02:35:28.516269 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:28.516277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:28.517234 (+ 957us) server_negotiation.cc:407] Connection header received
0620 02:35:28.517608 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:28.517614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:28.517691 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:28.517752 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:28.519504 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.520878 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.522699 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.523394 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.523546 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.526820 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:28.526830 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:28.526839 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:28.526891 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:28.530615 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:28.531469 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:28.531476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:28.531479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:28.531670 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:28.532148 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:28.532153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:28.532157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:28.532498 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:28.532675 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:28.533795 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0620 02:35:28.534050 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:35:28.535560 2004 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41785
I20260620 02:35:28.535992 2004 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:28.537215 2004 heartbeater.cc:511] Master 127.0.97.62:41785 requested a full tablet report, sending...
I20260620 02:35:28.539278 1517 ts_manager.cc:194] Registered new tserver with Master: 12ec6ffbae974b98aac5771bb02f9cb8 (127.0.97.3:38691)
I20260620 02:35:28.540249 1517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53133
I20260620 02:35:28.542759 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 1868
I20260620 02:35:28.542933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:35:28.543309 388 external_mini_cluster.cc:1428] Setting key 7c8934ca0c44331f32b6001e6d25dd3e
I20260620 02:35:28.546680 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:35:28 dist-test-slave-rpg6 krb5kdc[1472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:35:28 dist-test-slave-rpg6 krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:28.596964 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.581256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.0.97.62:41785)
0620 02:35:28.581500 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:28.581506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:28.581723 (+ 217us) server_negotiation.cc:407] Connection header received
0620 02:35:28.581943 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:28.581955 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:28.582022 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:28.582135 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:28.583598 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.584438 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.585811 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.586244 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.586392 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.591176 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:28.591182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:28.591186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:28.591231 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:28.594071 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:28.594868 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:28.594874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:28.594878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:28.595013 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:28.595413 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:28.595420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:28.595423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:28.595724 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:28.596034 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:28.596326 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 02:35:28.596578 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 02:35:28.601994 1517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:35:28.609401 1517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:35:28.646502 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.623666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.0.97.2:35435)
0620 02:35:28.637377 (+ 13711us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:28.637390 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:28.637429 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:35:28.637551 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:28.637559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:28.637839 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:28.638056 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:28.639900 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.641348 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.644022 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.645206 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.645362 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.645598 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:28.645958 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 02:35:28.646141 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13553,"thread_start_us":13409,"threads_started":1}
I20260620 02:35:28.646675 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.623350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35435 (local address 127.0.0.1:43146)
0620 02:35:28.636361 (+ 13011us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:28.636432 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:28.636732 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:28.638317 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:28.638329 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:28.638367 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:28.639674 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.639686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.641482 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.641487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.643866 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.643884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.645421 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.645427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.645720 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.645758 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:35:28.645897 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 02:35:28.646189 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11823,"mutex_wait_us":837,"thread_start_us":13603,"threads_started":1}
I20260620 02:35:28.647037 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.623373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38691 (local address 127.0.0.1:45298)
0620 02:35:28.636527 (+ 13154us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:28.636554 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:28.636731 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:28.638375 (+ 1644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:28.638379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:28.638435 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:28.638894 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.638910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.641073 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.641079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.643644 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.643658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.645482 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.645488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.645759 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.645796 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:35:28.645885 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 02:35:28.646184 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12954,"mutex_wait_us":100,"thread_start_us":11616,"threads_started":1}
I20260620 02:35:28.647478 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.637475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42585 (local address 127.0.0.1:56866)
0620 02:35:28.639580 (+ 2105us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:28.639606 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:28.639752 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:28.640577 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:28.640582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:28.640607 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:28.640933 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.640943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.642351 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.642359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.644585 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.644598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.646624 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.646631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.646821 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.646852 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:35:28.646931 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 02:35:28.647086 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1949,"mutex_wait_us":4,"thread_start_us":1791,"threads_started":1}
I20260620 02:35:28.648440 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.623476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.0.97.3:38691)
0620 02:35:28.637823 (+ 14347us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:28.637831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:28.637862 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:35:28.637957 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:28.637967 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:28.638177 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:28.639109 (+ 932us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:28.639476 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.640933 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.644024 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.645245 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.647491 (+ 2246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.647872 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:28.647941 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 02:35:28.648108 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14162,"thread_start_us":14979,"threads_started":1}
I20260620 02:35:28.649113 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.637719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.0.97.1:42585)
0620 02:35:28.639038 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:28.639048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:28.639900 (+ 852us) server_negotiation.cc:407] Connection header received
0620 02:35:28.640006 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:28.640014 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:28.640216 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:28.640395 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:28.641118 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.642209 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.644770 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.646217 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.646352 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.647822 (+ 1470us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:28.647948 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 02:35:28.648830 (+ 882us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":1008,"threads_started":1}
I20260620 02:35:28.650871 1794 tablet_service.cc:1467] Processing CreateTablet for tablet f1d0a644e2c6486c95d0fb0d01faf1c3 (DEFAULT_TABLE table=test-table [id=41cff5eb16094733ac01011c07e45352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:28.652251 1794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d0a644e2c6486c95d0fb0d01faf1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:28.652097 1934 tablet_service.cc:1467] Processing CreateTablet for tablet f1d0a644e2c6486c95d0fb0d01faf1c3 (DEFAULT_TABLE table=test-table [id=41cff5eb16094733ac01011c07e45352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:28.652248 1654 tablet_service.cc:1467] Processing CreateTablet for tablet f1d0a644e2c6486c95d0fb0d01faf1c3 (DEFAULT_TABLE table=test-table [id=41cff5eb16094733ac01011c07e45352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:28.653582 1934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d0a644e2c6486c95d0fb0d01faf1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:28.653606 1654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d0a644e2c6486c95d0fb0d01faf1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:28.665468 2023 tablet_bootstrap.cc:492] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: Bootstrap starting.
I20260620 02:35:28.668990 2023 tablet_bootstrap.cc:654] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:28.670162 2023 log.cc:824] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:28.673349 2024 tablet_bootstrap.cc:492] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0: Bootstrap starting.
I20260620 02:35:28.675767 2023 tablet_bootstrap.cc:492] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: No bootstrap required, opened a new log
I20260620 02:35:28.676111 2023 ts_tablet_manager.cc:1397] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260620 02:35:28.676400 2025 tablet_bootstrap.cc:492] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: Bootstrap starting.
I20260620 02:35:28.677240 2024 tablet_bootstrap.cc:654] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:28.678694 2024 log.cc:824] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:28.680487 2025 tablet_bootstrap.cc:654] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:28.681918 2025 log.cc:824] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:28.683231 2024 tablet_bootstrap.cc:492] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0: No bootstrap required, opened a new log
I20260620 02:35:28.683540 2024 ts_tablet_manager.cc:1397] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 02:35:28.684860 2025 tablet_bootstrap.cc:492] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: No bootstrap required, opened a new log
I20260620 02:35:28.685158 2025 ts_tablet_manager.cc:1397] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 02:35:28.687279 2023 raft_consensus.cc:348] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } }
I20260620 02:35:28.687809 2023 raft_consensus.cc:374] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:28.687968 2023 raft_consensus.cc:729] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a3265323bd485f88484359bb93cebf, State: Initialized, Role: FOLLOWER
I20260620 02:35:28.688720 2023 consensus_queue.cc:260] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } }
I20260620 02:35:28.689985 1864 heartbeater.cc:503] Master 127.0.97.62:41785 was elected leader, sending a full tablet report...
I20260620 02:35:28.690882 2023 ts_tablet_manager.cc:1428] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260620 02:35:28.694168 2025 raft_consensus.cc:348] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } }
I20260620 02:35:28.694612 2025 raft_consensus.cc:374] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:28.694768 2025 raft_consensus.cc:729] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ec6ffbae974b98aac5771bb02f9cb8, State: Initialized, Role: FOLLOWER
I20260620 02:35:28.695464 2024 raft_consensus.cc:348] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } }
I20260620 02:35:28.695515 2025 consensus_queue.cc:260] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } }
I20260620 02:35:28.695897 2024 raft_consensus.cc:374] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:28.696029 2024 raft_consensus.cc:729] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d614b3bbd54d98b3d6fda0678383a0, State: Initialized, Role: FOLLOWER
I20260620 02:35:28.696760 2024 consensus_queue.cc:260] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } }
I20260620 02:35:28.697284 2004 heartbeater.cc:503] Master 127.0.97.62:41785 was elected leader, sending a full tablet report...
I20260620 02:35:28.697661 2025 ts_tablet_manager.cc:1428] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 02:35:28.701349 2024 ts_tablet_manager.cc:1428] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0: Time spent starting tablet: real 0.018s user 0.014s sys 0.005s
W20260620 02:35:28.723330 1865 tablet.cc:2369] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:35:28.725495 2031 raft_consensus.cc:482] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:35:28.725802 2031 raft_consensus.cc:504] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } }
I20260620 02:35:28.727463 2031 leader_election.cc:288] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6a3265323bd485f88484359bb93cebf (127.0.97.2:35435), 12ec6ffbae974b98aac5771bb02f9cb8 (127.0.97.3:38691)
I20260620 02:35:28.738521 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.728696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52849 (local address 127.0.97.2:35435)
0620 02:35:28.728901 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:28.728908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:28.729723 (+ 815us) server_negotiation.cc:407] Connection header received
0620 02:35:28.729806 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:28.729810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:28.729929 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:28.730048 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:28.731027 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.732191 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.735747 (+ 3556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.737077 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.737255 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.737374 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:28.738084 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 02:35:28.738233 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:35:28.738518 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.728683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58621 (local address 127.0.97.3:38691)
0620 02:35:28.728901 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:28.728908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:28.729665 (+ 757us) server_negotiation.cc:407] Connection header received
0620 02:35:28.729935 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:28.729940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:28.730030 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:28.730125 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:28.731027 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.732191 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.735742 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.737075 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.737255 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.737374 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:28.738084 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 02:35:28.738233 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:35:28.738794 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.727820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35435 (local address 127.0.97.1:52849)
0620 02:35:28.729489 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:28.729509 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:28.729690 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:28.730071 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:28.730075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:28.730133 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:28.730783 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.730794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.732366 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.732371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.735484 (+ 3113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.735494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.737540 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.737545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.737777 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.737896 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 02:35:28.738014 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 02:35:28.738325 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1454,"mutex_wait_us":344,"thread_start_us":1321,"threads_started":1}
I20260620 02:35:28.738817 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.727910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38691 (local address 127.0.97.1:58621)
0620 02:35:28.729486 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:28.729509 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:28.729744 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:28.730141 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:28.730145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:28.730163 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:28.730722 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.730737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.732366 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.732371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.735464 (+ 3093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:28.735494 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.737523 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.737534 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:28.737780 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.737897 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 02:35:28.738011 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 02:35:28.738250 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1450,"mutex_wait_us":293,"spinlock_wait_cycles":100096,"thread_start_us":1180,"threads_started":1}
I20260620 02:35:28.740177 1819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d0a644e2c6486c95d0fb0d01faf1c3" candidate_uuid: "95d614b3bbd54d98b3d6fda0678383a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3265323bd485f88484359bb93cebf" is_pre_election: true
I20260620 02:35:28.740754 1819 raft_consensus.cc:2454] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d614b3bbd54d98b3d6fda0678383a0 in term 0.
I20260620 02:35:28.740180 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d0a644e2c6486c95d0fb0d01faf1c3" candidate_uuid: "95d614b3bbd54d98b3d6fda0678383a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" is_pre_election: true
I20260620 02:35:28.741212 1954 raft_consensus.cc:2454] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d614b3bbd54d98b3d6fda0678383a0 in term 0.
I20260620 02:35:28.741535 1605 leader_election.cc:302] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d614b3bbd54d98b3d6fda0678383a0, c6a3265323bd485f88484359bb93cebf; no voters:
I20260620 02:35:28.742048 2031 raft_consensus.cc:2781] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:35:28.742277 2031 raft_consensus.cc:482] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:35:28.742399 2031 raft_consensus.cc:3037] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:28.744781 2031 raft_consensus.cc:504] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } }
I20260620 02:35:28.745451 2031 leader_election.cc:288] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [CANDIDATE]: Term 1 election: Requested vote from peers c6a3265323bd485f88484359bb93cebf (127.0.97.2:35435), 12ec6ffbae974b98aac5771bb02f9cb8 (127.0.97.3:38691)
I20260620 02:35:28.746016 1819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d0a644e2c6486c95d0fb0d01faf1c3" candidate_uuid: "95d614b3bbd54d98b3d6fda0678383a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3265323bd485f88484359bb93cebf"
I20260620 02:35:28.746207 1819 raft_consensus.cc:3037] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:28.746467 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d0a644e2c6486c95d0fb0d01faf1c3" candidate_uuid: "95d614b3bbd54d98b3d6fda0678383a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ec6ffbae974b98aac5771bb02f9cb8"
I20260620 02:35:28.746712 1954 raft_consensus.cc:3037] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:28.748667 1819 raft_consensus.cc:2454] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d614b3bbd54d98b3d6fda0678383a0 in term 1.
I20260620 02:35:28.749145 1605 leader_election.cc:302] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d614b3bbd54d98b3d6fda0678383a0, c6a3265323bd485f88484359bb93cebf; no voters:
I20260620 02:35:28.749184 1954 raft_consensus.cc:2454] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d614b3bbd54d98b3d6fda0678383a0 in term 1.
I20260620 02:35:28.749495 2031 raft_consensus.cc:2781] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:28.749996 2031 raft_consensus.cc:686] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 LEADER]: Becoming Leader. State: Replica: 95d614b3bbd54d98b3d6fda0678383a0, State: Running, Role: LEADER
I20260620 02:35:28.750583 2031 consensus_queue.cc:237] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } }
I20260620 02:35:28.755405 1517 catalog_manager.cc:5495] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95d614b3bbd54d98b3d6fda0678383a0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "95d614b3bbd54d98b3d6fda0678383a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } health_report { overall_health: HEALTHY } } }
I20260620 02:35:28.788829 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:28.782944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.0.97.1:42585)
0620 02:35:28.783160 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:28.783169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:28.783191 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:35:28.783612 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:28.783616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:28.783701 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:28.783908 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:28.784282 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.785009 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.786369 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:28.786797 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:28.786954 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:28.787484 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:28.788150 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:28.788303 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:28.788444 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 02:35:28.788533 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 02:35:28.790601 2005 tablet.cc:2369] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:35:28.797631 1954 raft_consensus.cc:1261] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 1 FOLLOWER]: Refusing update from remote peer 95d614b3bbd54d98b3d6fda0678383a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:28.797665 1819 raft_consensus.cc:1261] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf [term 1 FOLLOWER]: Refusing update from remote peer 95d614b3bbd54d98b3d6fda0678383a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:28.798820 2034 consensus_queue.cc:1035] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:28.799300 2031 consensus_queue.cc:1035] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:28.813138 2037 mvcc.cc:205] Tried to move back new op lower bound from 7298756316343193600 to 7298756316166500352. Current Snapshot: MvccSnapshot[applied={T|T < 7298756316343193600}]
I20260620 02:35:28.832424 1516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:28.832791 1516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:28.837103 1516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c: Sending DeleteTablet for 3 replicas of tablet f1d0a644e2c6486c95d0fb0d01faf1c3
I20260620 02:35:28.838474 1934 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d0a644e2c6486c95d0fb0d01faf1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45298
I20260620 02:35:28.838577 1794 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d0a644e2c6486c95d0fb0d01faf1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43146
I20260620 02:35:28.839592 1654 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d0a644e2c6486c95d0fb0d01faf1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56866
I20260620 02:35:28.839910 1516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:35:28.840948 1516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:35:28.852319 1933 tablet_service.cc:1467] Processing CreateTablet for tablet f2854a45e93f4189968ceed1d46ddf21 (DEFAULT_TABLE table=test-table [id=ad3395ed8d2d49769977f27d0bdcf179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:28.853120 1933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2854a45e93f4189968ceed1d46ddf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:28.857900 1653 tablet_service.cc:1467] Processing CreateTablet for tablet f2854a45e93f4189968ceed1d46ddf21 (DEFAULT_TABLE table=test-table [id=ad3395ed8d2d49769977f27d0bdcf179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:28.857447 1793 tablet_service.cc:1467] Processing CreateTablet for tablet f2854a45e93f4189968ceed1d46ddf21 (DEFAULT_TABLE table=test-table [id=ad3395ed8d2d49769977f27d0bdcf179]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:28.858557 1793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2854a45e93f4189968ceed1d46ddf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:28.858599 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2854a45e93f4189968ceed1d46ddf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:28.865902 2023 tablet_bootstrap.cc:492] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf: Bootstrap starting.
I20260620 02:35:28.868398 2023 tablet_bootstrap.cc:654] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:28.870941 2023 tablet_bootstrap.cc:492] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf: No bootstrap required, opened a new log
I20260620 02:35:28.871124 2023 ts_tablet_manager.cc:1397] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 02:35:28.872095 2023 raft_consensus.cc:348] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:28.872454 2023 raft_consensus.cc:374] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:28.872653 2023 raft_consensus.cc:729] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a3265323bd485f88484359bb93cebf, State: Initialized, Role: FOLLOWER
I20260620 02:35:28.873140 2023 consensus_queue.cc:260] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:28.874583 2023 ts_tablet_manager.cc:1428] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 02:35:28.880892 2024 tablet_bootstrap.cc:492] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0: Bootstrap starting.
I20260620 02:35:28.881431 2025 tablet_bootstrap.cc:492] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8: Bootstrap starting.
I20260620 02:35:28.883226 2024 tablet_bootstrap.cc:654] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:28.883285 2025 tablet_bootstrap.cc:654] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:28.886626 2025 tablet_bootstrap.cc:492] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8: No bootstrap required, opened a new log
I20260620 02:35:28.888374 2049 tablet_replica.cc:330] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0: stopping tablet replica
I20260620 02:35:28.889405 2025 ts_tablet_manager.cc:1397] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260620 02:35:28.890223 2049 raft_consensus.cc:2229] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:35:28.890576 2025 raft_consensus.cc:348] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:28.891000 2025 raft_consensus.cc:374] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:28.891165 2025 raft_consensus.cc:729] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ec6ffbae974b98aac5771bb02f9cb8, State: Initialized, Role: FOLLOWER
I20260620 02:35:28.891655 2025 consensus_queue.cc:260] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:28.893337 2048 tablet_replica.cc:330] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: stopping tablet replica
I20260620 02:35:28.894300 2048 raft_consensus.cc:2229] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:28.895064 2049 raft_consensus.cc:2258] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:28.896203 2048 raft_consensus.cc:2258] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:28.896253 2025 ts_tablet_manager.cc:1428] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260620 02:35:28.897363 2047 tablet_replica.cc:330] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: stopping tablet replica
I20260620 02:35:28.897847 2047 raft_consensus.cc:2229] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:28.898347 2047 raft_consensus.cc:2258] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:28.898485 2024 tablet_bootstrap.cc:492] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0: No bootstrap required, opened a new log
I20260620 02:35:28.898692 2024 ts_tablet_manager.cc:1397] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.006s
I20260620 02:35:28.900009 2024 raft_consensus.cc:348] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:28.900328 2024 raft_consensus.cc:374] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:28.900481 2024 raft_consensus.cc:729] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d614b3bbd54d98b3d6fda0678383a0, State: Initialized, Role: FOLLOWER
I20260620 02:35:28.900817 2024 consensus_queue.cc:260] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:28.901419 2048 ts_tablet_manager.cc:1905] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:28.901499 2024 ts_tablet_manager.cc:1428] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 02:35:28.903115 2049 ts_tablet_manager.cc:1905] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:28.913038 2047 ts_tablet_manager.cc:1905] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:28.993360 2049 ts_tablet_manager.cc:1918] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:35:28.993582 2049 log.cc:1196] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/f1d0a644e2c6486c95d0fb0d01faf1c3
I20260620 02:35:28.994218 2049 ts_tablet_manager.cc:1939] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 95d614b3bbd54d98b3d6fda0678383a0: Deleting consensus metadata
I20260620 02:35:28.994210 2048 ts_tablet_manager.cc:1918] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:35:28.994379 2048 log.cc:1196] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/f1d0a644e2c6486c95d0fb0d01faf1c3
I20260620 02:35:28.994863 2048 ts_tablet_manager.cc:1939] T f1d0a644e2c6486c95d0fb0d01faf1c3 P c6a3265323bd485f88484359bb93cebf: Deleting consensus metadata
I20260620 02:35:29.089231 2047 ts_tablet_manager.cc:1918] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:35:29.089380 1502 catalog_manager.cc:4841] TS c6a3265323bd485f88484359bb93cebf (127.0.97.2:35435): tablet f1d0a644e2c6486c95d0fb0d01faf1c3 (table test-table [id=41cff5eb16094733ac01011c07e45352]) successfully deleted
I20260620 02:35:29.089594 2047 log.cc:1196] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/f1d0a644e2c6486c95d0fb0d01faf1c3
I20260620 02:35:29.089797 1502 catalog_manager.cc:4841] TS 95d614b3bbd54d98b3d6fda0678383a0 (127.0.97.1:42585): tablet f1d0a644e2c6486c95d0fb0d01faf1c3 (table test-table [id=41cff5eb16094733ac01011c07e45352]) successfully deleted
I20260620 02:35:29.090265 2047 ts_tablet_manager.cc:1939] T f1d0a644e2c6486c95d0fb0d01faf1c3 P 12ec6ffbae974b98aac5771bb02f9cb8: Deleting consensus metadata
I20260620 02:35:29.118862 1503 catalog_manager.cc:4841] TS 12ec6ffbae974b98aac5771bb02f9cb8 (127.0.97.3:38691): tablet f1d0a644e2c6486c95d0fb0d01faf1c3 (table test-table [id=41cff5eb16094733ac01011c07e45352]) successfully deleted
W20260620 02:35:29.147049 1725 tablet.cc:2369] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:35:29.176857 2029 raft_consensus.cc:482] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:35:29.177109 2029 raft_consensus.cc:504] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.179092 2029 leader_election.cc:288] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95d614b3bbd54d98b3d6fda0678383a0 (127.0.97.1:42585), 12ec6ffbae974b98aac5771bb02f9cb8 (127.0.97.3:38691)
I20260620 02:35:29.188545 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.179541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35259 (local address 127.0.97.3:38691)
0620 02:35:29.179739 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.179747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.180348 (+ 601us) server_negotiation.cc:407] Connection header received
0620 02:35:29.180692 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.180699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.180790 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.180859 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:29.181745 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.183054 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.185813 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.187066 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.187220 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.187336 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.188081 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.188259 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:35:29.188568 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.179450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38691 (local address 127.0.97.2:35259)
0620 02:35:29.180227 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:29.180250 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:29.180491 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:29.181073 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:29.181078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:29.181104 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:29.181554 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.181574 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.183200 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.183208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.185583 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.185601 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.187525 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.187532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.187768 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.187815 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 02:35:29.188014 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 02:35:29.188185 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":1124,"threads_started":1}
I20260620 02:35:29.188818 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.179931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56929 (local address 127.0.97.1:42585)
0620 02:35:29.180169 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.180177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.180986 (+ 809us) server_negotiation.cc:407] Connection header received
0620 02:35:29.181192 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.181202 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.181270 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.181372 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:29.182514 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.183665 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.186522 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.187574 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.187765 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.187914 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.188309 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.188456 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:35:29.189564 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.179472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42585 (local address 127.0.97.2:56929)
0620 02:35:29.180871 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:29.180889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:29.181038 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:29.181413 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:29.181419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:29.181445 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:29.182284 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.182300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.184576 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.184582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.186350 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.186362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.187854 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.187861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.188088 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.188121 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:35:29.188998 (+ 877us) client_negotiation.cc:240] Negotiation successful
0620 02:35:29.189179 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"thread_start_us":713,"threads_started":1}
I20260620 02:35:29.189965 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2854a45e93f4189968ceed1d46ddf21" candidate_uuid: "c6a3265323bd485f88484359bb93cebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" is_pre_election: true
I20260620 02:35:29.190263 1954 raft_consensus.cc:2454] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6a3265323bd485f88484359bb93cebf in term 0.
I20260620 02:35:29.190848 1746 leader_election.cc:302] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12ec6ffbae974b98aac5771bb02f9cb8, c6a3265323bd485f88484359bb93cebf; no voters:
I20260620 02:35:29.191385 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2854a45e93f4189968ceed1d46ddf21" candidate_uuid: "c6a3265323bd485f88484359bb93cebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d614b3bbd54d98b3d6fda0678383a0" is_pre_election: true
I20260620 02:35:29.191895 1674 raft_consensus.cc:2454] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6a3265323bd485f88484359bb93cebf in term 0.
I20260620 02:35:29.192865 2029 raft_consensus.cc:2781] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:35:29.193009 2029 raft_consensus.cc:482] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:35:29.193107 2029 raft_consensus.cc:3037] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:29.274914 2029 raft_consensus.cc:504] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.275813 2029 leader_election.cc:288] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [CANDIDATE]: Term 1 election: Requested vote from peers 95d614b3bbd54d98b3d6fda0678383a0 (127.0.97.1:42585), 12ec6ffbae974b98aac5771bb02f9cb8 (127.0.97.3:38691)
I20260620 02:35:29.276391 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2854a45e93f4189968ceed1d46ddf21" candidate_uuid: "c6a3265323bd485f88484359bb93cebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d614b3bbd54d98b3d6fda0678383a0"
I20260620 02:35:29.276660 1674 raft_consensus.cc:3037] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:29.276695 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2854a45e93f4189968ceed1d46ddf21" candidate_uuid: "c6a3265323bd485f88484359bb93cebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ec6ffbae974b98aac5771bb02f9cb8"
I20260620 02:35:29.276904 1954 raft_consensus.cc:3037] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:29.279968 1954 raft_consensus.cc:2454] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6a3265323bd485f88484359bb93cebf in term 1.
I20260620 02:35:29.280552 1746 leader_election.cc:302] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12ec6ffbae974b98aac5771bb02f9cb8, c6a3265323bd485f88484359bb93cebf; no voters:
I20260620 02:35:29.280949 1674 raft_consensus.cc:2454] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6a3265323bd485f88484359bb93cebf in term 1.
I20260620 02:35:29.281088 2029 raft_consensus.cc:2781] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:29.282513 2029 raft_consensus.cc:686] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 1 LEADER]: Becoming Leader. State: Replica: c6a3265323bd485f88484359bb93cebf, State: Running, Role: LEADER
I20260620 02:35:29.283485 2029 consensus_queue.cc:237] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.287485 1516 catalog_manager.cc:5495] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf reported cstate change: term changed from 0 to 1, leader changed from <none> to c6a3265323bd485f88484359bb93cebf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c6a3265323bd485f88484359bb93cebf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } health_report { overall_health: UNKNOWN } } }
Jun 20 02:35:29 dist-test-slave-rpg6 krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:29.383586 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.366408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41785 (local address 127.0.0.1:44436)
0620 02:35:29.368001 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:29.368018 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:29.368223 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:29.368568 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:29.368572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:29.369423 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:29.369737 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.369750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.370823 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.370828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.372210 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.372223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.372949 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.372955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.373132 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.373860 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:29.373878 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:29.376491 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:29.380752 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:29.380766 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:29.380773 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:29.381248 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:29.381716 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:29.381719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:29.381722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:29.381870 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:29.382692 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:29.382699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:29.382867 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 02:35:29.382998 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 02:35:29.383267 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1485,"spinlock_wait_cycles":8576,"thread_start_us":1279,"threads_started":1}
I20260620 02:35:29.383730 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.366434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.0.97.62:41785)
0620 02:35:29.367136 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.367143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.368135 (+ 992us) server_negotiation.cc:407] Connection header received
0620 02:35:29.368363 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.368367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.368426 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.368548 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:29.369911 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.370683 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.372355 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.372780 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.372929 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.376724 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:29.376730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:29.376735 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:29.376765 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:29.380568 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.381406 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.381415 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.381422 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.381580 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.382210 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.382218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.382226 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.382547 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:29.382678 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.383197 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.383360 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":394,"threads_started":1}
I20260620 02:35:29.389395 1516 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44436:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:35:29.390544 1516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:35:29.398296 1653 tablet_service.cc:1467] Processing CreateTablet for tablet 8dd6dc12635a4c1186391cef3abe64ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc930d9b9a14fbe8aced7cc7da02781]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:35:29.398397 1794 tablet_service.cc:1467] Processing CreateTablet for tablet 8dd6dc12635a4c1186391cef3abe64ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc930d9b9a14fbe8aced7cc7da02781]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:35:29.398694 1934 tablet_service.cc:1467] Processing CreateTablet for tablet 8dd6dc12635a4c1186391cef3abe64ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc930d9b9a14fbe8aced7cc7da02781]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:35:29.398881 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dd6dc12635a4c1186391cef3abe64ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:29.398967 1794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dd6dc12635a4c1186391cef3abe64ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:29.399188 1934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dd6dc12635a4c1186391cef3abe64ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:29.409646 2066 tablet_bootstrap.cc:492] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8: Bootstrap starting.
I20260620 02:35:29.410488 2064 tablet_bootstrap.cc:492] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: Bootstrap starting.
I20260620 02:35:29.412209 2066 tablet_bootstrap.cc:654] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:29.413281 2064 tablet_bootstrap.cc:654] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:29.417229 2065 tablet_bootstrap.cc:492] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf: Bootstrap starting.
I20260620 02:35:29.418249 2066 tablet_bootstrap.cc:492] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8: No bootstrap required, opened a new log
I20260620 02:35:29.421687 2065 tablet_bootstrap.cc:654] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:29.423103 2066 ts_tablet_manager.cc:1397] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260620 02:35:29.424265 2066 raft_consensus.cc:348] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.424523 2066 raft_consensus.cc:374] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:29.424642 2066 raft_consensus.cc:729] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ec6ffbae974b98aac5771bb02f9cb8, State: Initialized, Role: FOLLOWER
I20260620 02:35:29.424917 2066 consensus_queue.cc:260] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.425751 2064 tablet_bootstrap.cc:492] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: No bootstrap required, opened a new log
I20260620 02:35:29.425781 2065 tablet_bootstrap.cc:492] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf: No bootstrap required, opened a new log
I20260620 02:35:29.425971 2065 ts_tablet_manager.cc:1397] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260620 02:35:29.425989 2064 ts_tablet_manager.cc:1397] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260620 02:35:29.426607 2065 raft_consensus.cc:348] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.426921 2065 raft_consensus.cc:374] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:29.427302 2065 raft_consensus.cc:729] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a3265323bd485f88484359bb93cebf, State: Initialized, Role: FOLLOWER
I20260620 02:35:29.427681 2064 raft_consensus.cc:348] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.428098 2064 raft_consensus.cc:374] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:29.428236 2064 raft_consensus.cc:729] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d614b3bbd54d98b3d6fda0678383a0, State: Initialized, Role: FOLLOWER
I20260620 02:35:29.428128 2065 consensus_queue.cc:260] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.428529 2064 consensus_queue.cc:260] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.428723 2029 tablet_replica.cc:439] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } }
I20260620 02:35:29.429033 2029 tablet_replica.cc:442] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:29.430773 2065 ts_tablet_manager.cc:1428] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 02:35:29.431273 2071 tablet_replica.cc:439] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } }
I20260620 02:35:29.432111 2064 ts_tablet_manager.cc:1428] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 02:35:29.432332 2071 tablet_replica.cc:442] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:29.433214 2066 ts_tablet_manager.cc:1428] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 02:35:29.433941 2070 tablet_replica.cc:439] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } }
I20260620 02:35:29.434652 2070 tablet_replica.cc:442] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:29.444535 2071 raft_consensus.cc:482] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:35:29.444734 2071 raft_consensus.cc:504] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.445509 2071 leader_election.cc:288] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6a3265323bd485f88484359bb93cebf (127.0.97.2:35435), 12ec6ffbae974b98aac5771bb02f9cb8 (127.0.97.3:38691)
I20260620 02:35:29.446066 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd6dc12635a4c1186391cef3abe64ac" candidate_uuid: "95d614b3bbd54d98b3d6fda0678383a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" is_pre_election: true
I20260620 02:35:29.446079 1819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd6dc12635a4c1186391cef3abe64ac" candidate_uuid: "95d614b3bbd54d98b3d6fda0678383a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3265323bd485f88484359bb93cebf" is_pre_election: true
I20260620 02:35:29.446352 1954 raft_consensus.cc:2454] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d614b3bbd54d98b3d6fda0678383a0 in term 0.
I20260620 02:35:29.446364 1819 raft_consensus.cc:2454] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d614b3bbd54d98b3d6fda0678383a0 in term 0.
I20260620 02:35:29.447288 1606 leader_election.cc:302] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12ec6ffbae974b98aac5771bb02f9cb8, 95d614b3bbd54d98b3d6fda0678383a0; no voters:
I20260620 02:35:29.447626 2071 raft_consensus.cc:2781] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:35:29.447772 2071 raft_consensus.cc:482] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:35:29.447916 2071 raft_consensus.cc:3037] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:29.482297 2071 raft_consensus.cc:504] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.483315 2071 leader_election.cc:288] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [CANDIDATE]: Term 1 election: Requested vote from peers c6a3265323bd485f88484359bb93cebf (127.0.97.2:35435), 12ec6ffbae974b98aac5771bb02f9cb8 (127.0.97.3:38691)
I20260620 02:35:29.484077 1819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd6dc12635a4c1186391cef3abe64ac" candidate_uuid: "95d614b3bbd54d98b3d6fda0678383a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a3265323bd485f88484359bb93cebf"
I20260620 02:35:29.484334 1819 raft_consensus.cc:3037] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:29.484254 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd6dc12635a4c1186391cef3abe64ac" candidate_uuid: "95d614b3bbd54d98b3d6fda0678383a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ec6ffbae974b98aac5771bb02f9cb8"
I20260620 02:35:29.484499 1954 raft_consensus.cc:3037] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:29.487499 1819 raft_consensus.cc:2454] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d614b3bbd54d98b3d6fda0678383a0 in term 1.
I20260620 02:35:29.487483 1954 raft_consensus.cc:2454] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d614b3bbd54d98b3d6fda0678383a0 in term 1.
I20260620 02:35:29.488128 1605 leader_election.cc:302] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12ec6ffbae974b98aac5771bb02f9cb8, 95d614b3bbd54d98b3d6fda0678383a0, c6a3265323bd485f88484359bb93cebf; no voters:
I20260620 02:35:29.488569 2071 raft_consensus.cc:2781] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:29.489008 2071 raft_consensus.cc:686] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 LEADER]: Becoming Leader. State: Replica: 95d614b3bbd54d98b3d6fda0678383a0, State: Running, Role: LEADER
I20260620 02:35:29.489344 2071 consensus_queue.cc:237] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } }
I20260620 02:35:29.491511 2072 tablet_replica.cc:439] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: TxnStatusTablet state changed. Reason: New leader 95d614b3bbd54d98b3d6fda0678383a0. Latest consensus state: current_term: 1 leader_uuid: "95d614b3bbd54d98b3d6fda0678383a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } }
I20260620 02:35:29.491797 2072 tablet_replica.cc:442] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:35:29.492915 2074 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:35:29.492525 1515 catalog_manager.cc:5495] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95d614b3bbd54d98b3d6fda0678383a0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "95d614b3bbd54d98b3d6fda0678383a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } health_report { overall_health: UNKNOWN } } }
I20260620 02:35:29.493201 2074 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:35:29.529552 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.522943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.0.97.1:42585)
0620 02:35:29.523126 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.523132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.523199 (+ 67us) server_negotiation.cc:407] Connection header received
0620 02:35:29.523462 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.523467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.523565 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.523652 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:29.524117 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.525030 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.527106 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.527709 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.527919 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.528460 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:29.528589 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:29.528674 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.528862 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.528935 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:35:29.529767 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.522780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42585 (local address 127.0.0.1:56874)
0620 02:35:29.523057 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:29.523080 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:29.523243 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:29.523720 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:29.523726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:29.523740 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:35:29.523956 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.523968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.525196 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.525205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.526915 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.526932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.527920 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.527927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.528148 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.528236 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:35:29.528706 (+ 470us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:35:29.528730 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:35:29.528824 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 02:35:29.529004 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 02:35:29.537089 1954 raft_consensus.cc:1261] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8 [term 1 FOLLOWER]: Refusing update from remote peer 95d614b3bbd54d98b3d6fda0678383a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:29.537418 1819 raft_consensus.cc:1261] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf [term 1 FOLLOWER]: Refusing update from remote peer 95d614b3bbd54d98b3d6fda0678383a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:29.538043 2071 consensus_queue.cc:1035] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:29.538406 2072 consensus_queue.cc:1035] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:29.555873 2029 tablet_replica.cc:439] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf: TxnStatusTablet state changed. Reason: New leader 95d614b3bbd54d98b3d6fda0678383a0. Latest consensus state: current_term: 1 leader_uuid: "95d614b3bbd54d98b3d6fda0678383a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } }
I20260620 02:35:29.556272 2029 tablet_replica.cc:442] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:29.558238 2070 tablet_replica.cc:439] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8: TxnStatusTablet state changed. Reason: New leader 95d614b3bbd54d98b3d6fda0678383a0. Latest consensus state: current_term: 1 leader_uuid: "95d614b3bbd54d98b3d6fda0678383a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } }
I20260620 02:35:29.558511 2070 tablet_replica.cc:442] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:29.561143 2071 tablet_replica.cc:439] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95d614b3bbd54d98b3d6fda0678383a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } }
I20260620 02:35:29.561478 2071 tablet_replica.cc:442] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:35:29.561431 2078 mvcc.cc:205] Tried to move back new op lower bound from 7298756319376158720 to 7298756319190990848. Current Snapshot: MvccSnapshot[applied={T|T < 7298756319376158720}]
I20260620 02:35:29.567444 2072 tablet_replica.cc:439] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95d614b3bbd54d98b3d6fda0678383a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } }
I20260620 02:35:29.567775 2072 tablet_replica.cc:442] T 8dd6dc12635a4c1186391cef3abe64ac P 95d614b3bbd54d98b3d6fda0678383a0: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:35:29.569012 2079 mvcc.cc:205] Tried to move back new op lower bound from 7298756319376158720 to 7298756319190990848. Current Snapshot: MvccSnapshot[applied={T|T < 7298756319376158720}]
I20260620 02:35:29.569835 2029 tablet_replica.cc:439] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95d614b3bbd54d98b3d6fda0678383a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } }
I20260620 02:35:29.570179 2029 tablet_replica.cc:442] T 8dd6dc12635a4c1186391cef3abe64ac P c6a3265323bd485f88484359bb93cebf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:29.571487 2070 tablet_replica.cc:439] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95d614b3bbd54d98b3d6fda0678383a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 } } peers { permanent_uuid: "c6a3265323bd485f88484359bb93cebf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35435 } } peers { permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 } } }
I20260620 02:35:29.571780 2070 tablet_replica.cc:442] T 8dd6dc12635a4c1186391cef3abe64ac P 12ec6ffbae974b98aac5771bb02f9cb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:29.597433 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.576411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.0.97.2:35435)
0620 02:35:29.589386 (+ 12975us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.589393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.590406 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 02:35:29.590542 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.590547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.590669 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.590836 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:29.591489 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.592339 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.594433 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.595100 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.595233 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.595814 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:29.596614 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:29.596733 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.596898 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.596972 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12274,"spinlock_wait_cycles":291712,"thread_start_us":12594,"threads_started":1}
I20260620 02:35:29.615412 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.609544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53213 (local address 127.0.97.1:42585)
0620 02:35:29.609728 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.609733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.610167 (+ 434us) server_negotiation.cc:407] Connection header received
0620 02:35:29.610403 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.610415 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.610487 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.610587 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:29.611114 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.611912 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.613489 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.613978 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.614149 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.614591 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:29.614721 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:29.614824 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.614970 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.615055 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 02:35:29.615474 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.609451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42585 (local address 127.0.97.2:53213)
0620 02:35:29.610051 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:29.610073 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:29.610250 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:29.610624 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:29.610629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:29.610640 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:35:29.610956 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.610964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.612058 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.612064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.613357 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.613375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.614172 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.614179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.614330 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.614393 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:35:29.614836 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:35:29.614849 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:35:29.614949 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 02:35:29.615091 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":312,"threads_started":1}
I20260620 02:35:29.626019 1674 raft_consensus.cc:1261] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 FOLLOWER]: Refusing update from remote peer c6a3265323bd485f88484359bb93cebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:29.626035 1954 raft_consensus.cc:1261] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 1 FOLLOWER]: Refusing update from remote peer c6a3265323bd485f88484359bb93cebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:29.626789 2055 consensus_queue.cc:1035] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ec6ffbae974b98aac5771bb02f9cb8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:29.627247 2029 consensus_queue.cc:1035] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "95d614b3bbd54d98b3d6fda0678383a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:29.678869 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.672144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44213 (local address 127.0.97.2:35435)
0620 02:35:29.672337 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.672345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.672810 (+ 465us) server_negotiation.cc:407] Connection header received
0620 02:35:29.673057 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.673062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.673148 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.673249 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:29.673838 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.674686 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.676806 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.677240 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.677390 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.678012 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:29.678161 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:29.678259 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.678429 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.678569 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 02:35:29.679046 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.672072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35435 (local address 127.0.97.1:44213)
0620 02:35:29.672696 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:29.672715 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:29.672905 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:29.673267 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:29.673271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:29.673282 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:35:29.673660 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.673673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.674809 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.675162 (+ 353us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.676631 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:29.676645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.677475 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.677482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:29.677685 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.677804 (+ 119us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:35:29.678279 (+ 475us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:35:29.678290 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:35:29.678382 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 02:35:29.678529 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":291,"threads_started":1}
I20260620 02:35:29.732340 1517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:29.732661 1517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:29.737820 1517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbf937f8a990411c95c5b920e8ec0b3c: Sending DeleteTablet for 3 replicas of tablet f2854a45e93f4189968ceed1d46ddf21
I20260620 02:35:29.738979 1934 tablet_service.cc:1514] Processing DeleteTablet for tablet f2854a45e93f4189968ceed1d46ddf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45298
I20260620 02:35:29.739440 1653 tablet_service.cc:1514] Processing DeleteTablet for tablet f2854a45e93f4189968ceed1d46ddf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56866
I20260620 02:35:29.739625 2099 tablet_replica.cc:330] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8: stopping tablet replica
I20260620 02:35:29.739614 1794 tablet_service.cc:1514] Processing DeleteTablet for tablet f2854a45e93f4189968ceed1d46ddf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43146
I20260620 02:35:29.739861 2099 raft_consensus.cc:2229] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:29.740175 2099 raft_consensus.cc:2258] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:29.748242 2099 ts_tablet_manager.cc:1905] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:29.757818 2101 tablet_replica.cc:330] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf: stopping tablet replica
I20260620 02:35:29.758875 2101 raft_consensus.cc:2229] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:35:29.759761 2100 tablet_replica.cc:330] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0: stopping tablet replica
I20260620 02:35:29.760878 2100 raft_consensus.cc:2229] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:29.761420 2100 raft_consensus.cc:2258] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:29.761670 2101 raft_consensus.cc:2258] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:29.764277 2101 ts_tablet_manager.cc:1905] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:29.765146 2100 ts_tablet_manager.cc:1905] T f2854a45e93f4189968ceed1d46ddf21 P 95d614b3bbd54d98b3d6fda0678383a0: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:35:29 dist-test-slave-rpg6 krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:35:29.784144 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.742278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.0.97.1:42585)
0620 02:35:29.742567 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.742576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.764649 (+ 22073us) server_negotiation.cc:407] Connection header received
0620 02:35:29.764860 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.764865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.764957 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.765070 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:29.767349 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.768625 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.771085 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.771911 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.772053 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.777136 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:29.777153 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:29.777175 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:29.777259 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:29.781048 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.781851 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.781868 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.781870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.782002 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.782411 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.782419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.782422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.782824 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:29.783145 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.783465 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.783752 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260620 02:35:29.785082 1719 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56880
I20260620 02:35:29.791816 2099 ts_tablet_manager.cc:1918] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:35:29.792054 2099 log.cc:1196] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/f2854a45e93f4189968ceed1d46ddf21
I20260620 02:35:29.793367 2099 ts_tablet_manager.cc:1939] T f2854a45e93f4189968ceed1d46ddf21 P 12ec6ffbae974b98aac5771bb02f9cb8: Deleting consensus metadata
I20260620 02:35:29.818671 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.797594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.0.97.62:41785)
0620 02:35:29.797834 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.797841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.804204 (+ 6363us) server_negotiation.cc:407] Connection header received
0620 02:35:29.804452 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.804459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.804587 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.804650 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:29.806265 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.807101 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.808452 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.808906 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.809050 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.811761 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:29.811767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:29.811770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:29.811809 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:29.815815 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.816661 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.816669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.816673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.816816 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.817188 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.817194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.817198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.817500 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:29.817761 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.818084 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.818341 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 02:35:29.819880 1517 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44448
I20260620 02:35:29.823653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 1588
I20260620 02:35:29.841305 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.824859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.0.97.1:42585)
0620 02:35:29.825056 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.825063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.825085 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:35:29.825233 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.825238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.825326 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.825390 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:29.826812 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.829124 (+ 2312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.830721 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.831297 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.831463 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.833836 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:29.833844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:29.833847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:29.833885 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:29.837407 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.838637 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.838643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.838647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.838827 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.839222 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.839235 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.839239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.839517 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:29.839804 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.840134 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.840391 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 02:35:29.841977 1719 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56888
W20260620 02:35:29.842713 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 1588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:35:29.842912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 1588
I20260620 02:35:29.868592 1503 catalog_manager.cc:4841] TS 12ec6ffbae974b98aac5771bb02f9cb8 (127.0.97.3:38691): tablet f2854a45e93f4189968ceed1d46ddf21 (table test-table [id=ad3395ed8d2d49769977f27d0bdcf179]) successfully deleted
I20260620 02:35:29.881974 2101 ts_tablet_manager.cc:1918] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:35:29.882185 2101 log.cc:1196] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/f2854a45e93f4189968ceed1d46ddf21
I20260620 02:35:29.882705 2101 ts_tablet_manager.cc:1939] T f2854a45e93f4189968ceed1d46ddf21 P c6a3265323bd485f88484359bb93cebf: Deleting consensus metadata
I20260620 02:35:29.886732 1502 catalog_manager.cc:4841] TS c6a3265323bd485f88484359bb93cebf (127.0.97.2:35435): tablet f2854a45e93f4189968ceed1d46ddf21 (table test-table [id=ad3395ed8d2d49769977f27d0bdcf179]) successfully deleted
I20260620 02:35:29.905237 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 1728
W20260620 02:35:29.905141 1502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42585 (error 108)
I20260620 02:35:29.908034 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.906732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42585 (local address 127.0.0.1:56902)
0620 02:35:29.907450 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:29.907584 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":455,"threads_started":1}
W20260620 02:35:29.908361 1502 catalog_manager.cc:4584] TS 95d614b3bbd54d98b3d6fda0678383a0 (127.0.97.1:42585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2854a45e93f4189968ceed1d46ddf21: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42585: connect: Connection refused (error 111)
Jun 20 02:35:29 dist-test-slave-rpg6 krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:35:29.924069 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.906247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.0.97.2:35435)
0620 02:35:29.906454 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.906462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.906486 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:35:29.906623 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.906628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.906733 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.906864 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:29.908646 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.909812 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.911374 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.912089 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.912224 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.916858 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:29.916875 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:29.916939 (+ 64us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:29.916989 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:29.920514 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.921518 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.921529 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.921532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.921674 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.922151 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.922156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.922158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.922633 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:29.922926 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.923398 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.923620 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 02:35:29.924929 1859 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43152
W20260620 02:35:29.925722 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 1728: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:35:29.925935 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 1728
I20260620 02:35:29.945323 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 1868
Jun 20 02:35:29 dist-test-slave-rpg6 krb5kdc[1472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:35:29.962389 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.961833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42585 (local address 127.0.0.1:56918)
0620 02:35:29.962066 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:29.962163 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 02:35:29.963233 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.946317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45300 (local address 127.0.97.3:38691)
0620 02:35:29.946816 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:29.946821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:29.946837 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:35:29.946890 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:29.946894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:29.946958 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:29.947101 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:29.948535 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.949332 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.950660 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:29.951154 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:29.951306 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:29.956528 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:29.956545 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:29.956562 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:29.956604 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:29.959542 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.960685 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.960694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.960697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.960863 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:29.961398 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:29.961404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:29.961407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:29.961909 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:29.962227 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:29.962589 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 02:35:29.962830 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":248,"threads_started":1}
I20260620 02:35:29.963568 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:29.963123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42585 (local address 127.0.0.1:56932)
0620 02:35:29.963303 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:29.963408 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:35:29.964298 1999 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45300
W20260620 02:35:29.965018 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 1868: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:35:29.965193 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 1868
I20260620 02:35:29.984974 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 1485
I20260620 02:35:29.986435 1557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:30.111006 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 1485
I20260620 02:35:30.111049 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:30.110051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42585 (local address 127.0.0.1:56944)
0620 02:35:30.110802 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:30.110905 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603}
2026-06-20T02:35:30Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5250 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:35:30 dist-test-slave-rpg6 krb5kdc[2113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:35:30 dist-test-slave-rpg6 krb5kdc[2113](info): set up 2 sockets
Jun 20 02:35:30 dist-test-slave-rpg6 krb5kdc[2113](info): commencing operation
krb5kdc: starting...
W20260620 02:35:31.256218 388 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:35:31 dist-test-slave-rpg6 krb5kdc[2113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:35:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:35:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:35:31.460265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33311
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:31.807773 2126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:31.808159 2126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:31.808320 2126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:31.816030 2126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:35:31.816180 2126 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:35:31.816222 2126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:31.816319 2126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:31.816397 2126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:35:31.816510 2126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:35:31.827286 2126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33311
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:31.828787 2126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:31.830245 2126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:31.843698 2134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:31.843892 2132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:31.843851 2131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:31.844913 2126 server_base.cc:1034] running on GCE node
I20260620 02:35:31.845826 2126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:31.847563 2126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:31.848829 2126 hybrid_clock.cc:648] HybridClock initialized: now 1781922931848743 us; error 56 us; skew 500 ppm
I20260620 02:35:31.849247 2126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:31 dist-test-slave-rpg6 krb5kdc[2113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:31.852944 2126 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:35:31.854678 2126 webserver.cc:460] Webserver started at http://127.0.97.62:44689/ using document root <none> and password file <none>
I20260620 02:35:31.855451 2126 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:31.855551 2126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:31.855973 2126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:31.859202 2126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "0611fa41457c464c9ef4a48227f37cd5"
format_stamp: "Formatted at 2026-06-20 02:35:31 on dist-test-slave-rpg6"
server_key: "68272d5d671f778fbe4b6e7cca3494a2"
server_key_iv: "f09ea551f43e6ece701905361de92534"
server_key_version: "encryptionkey@0"
I20260620 02:35:31.860147 2126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "0611fa41457c464c9ef4a48227f37cd5"
format_stamp: "Formatted at 2026-06-20 02:35:31 on dist-test-slave-rpg6"
server_key: "68272d5d671f778fbe4b6e7cca3494a2"
server_key_iv: "f09ea551f43e6ece701905361de92534"
server_key_version: "encryptionkey@0"
I20260620 02:35:31.865792 2126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:35:31.869760 2141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:31.871337 2126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:35:31.871598 2126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "0611fa41457c464c9ef4a48227f37cd5"
format_stamp: "Formatted at 2026-06-20 02:35:31 on dist-test-slave-rpg6"
server_key: "68272d5d671f778fbe4b6e7cca3494a2"
server_key_iv: "f09ea551f43e6ece701905361de92534"
server_key_version: "encryptionkey@0"
I20260620 02:35:31.871907 2126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:31.892406 2126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:31.897430 2126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:31.897751 2126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:31.921888 2126 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33311
I20260620 02:35:31.921895 2202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33311 every 8 connection(s)
I20260620 02:35:31.923414 2126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:35:31.927480 2203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:31.930776 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 2126
I20260620 02:35:31.931008 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:35:31.931636 388 external_mini_cluster.cc:1428] Setting key 420d07774d355da594614456e01ebe88
I20260620 02:35:31.940610 2203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5: Bootstrap starting.
I20260620 02:35:31.944947 2203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:31.946516 2203 log.cc:824] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:35:31 dist-test-slave-rpg6 krb5kdc[2113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:31.950471 2203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5: No bootstrap required, opened a new log
I20260620 02:35:31.958741 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:31.933882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.0.97.62:33311)
0620 02:35:31.934734 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:31.934766 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:31.934812 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:35:31.936670 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:31.936696 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:31.937170 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:31.937637 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:31.939162 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:31.940634 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:31.942534 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:31.943294 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:31.943454 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:31.948772 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:31.948785 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:31.948807 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:31.948852 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:31.953241 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:31.954115 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:31.954125 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:31.954136 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:31.954393 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:31.954859 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:31.954865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:31.954868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:31.955714 (+ 846us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:31.956079 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:31.956562 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 02:35:31.957121 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":387,"threads_started":1}
I20260620 02:35:31.961606 2203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0611fa41457c464c9ef4a48227f37cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33311 } }
I20260620 02:35:31.962143 2203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:31.962266 2203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0611fa41457c464c9ef4a48227f37cd5, State: Initialized, Role: FOLLOWER
I20260620 02:35:31.962970 2203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0611fa41457c464c9ef4a48227f37cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33311 } }
I20260620 02:35:31.963238 2203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:35:31.963397 2203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:35:31.963577 2203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:31.966123 2203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0611fa41457c464c9ef4a48227f37cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33311 } }
I20260620 02:35:31.966785 2203 leader_election.cc:302] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0611fa41457c464c9ef4a48227f37cd5; no voters:
I20260620 02:35:31.967561 2203 leader_election.cc:288] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:35:31.967662 2208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:31.968531 2208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [term 1 LEADER]: Becoming Leader. State: Replica: 0611fa41457c464c9ef4a48227f37cd5, State: Running, Role: LEADER
I20260620 02:35:31.969125 2208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0611fa41457c464c9ef4a48227f37cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33311 } }
I20260620 02:35:31.969906 2203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:35:31.972563 2209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0611fa41457c464c9ef4a48227f37cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0611fa41457c464c9ef4a48227f37cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33311 } } }
I20260620 02:35:31.972728 2210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0611fa41457c464c9ef4a48227f37cd5. Latest consensus state: current_term: 1 leader_uuid: "0611fa41457c464c9ef4a48227f37cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0611fa41457c464c9ef4a48227f37cd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33311 } } }
I20260620 02:35:31.973042 2209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:31.973047 2210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:31.974282 2217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:35:31.984381 2217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:35:31.995321 2217 catalog_manager.cc:1347] Generated new cluster ID: bcf2bc0330a9410a96496298b4580bcc
I20260620 02:35:31.995458 2217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:35:32.010456 2217 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:35:32.011929 2217 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:35:32.021714 2217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5: Generated new TSK 0
I20260620 02:35:32.022933 2217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:35:32.081837 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33311
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:32.435936 2229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:32.436239 2229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:32.436342 2229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:32.443989 2229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:32.444118 2229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:32.444203 2229 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:35:32.444257 2229 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:35:32.444326 2229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:35:32.454634 2229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:32.456043 2229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:32.457403 2229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:32.472064 2234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:32.472178 2237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:32.472020 2235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:32.473690 2229 server_base.cc:1034] running on GCE node
I20260620 02:35:32.474483 2229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:32.475725 2229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:32.477012 2229 hybrid_clock.cc:648] HybridClock initialized: now 1781922932476965 us; error 40 us; skew 500 ppm
I20260620 02:35:32.477396 2229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:32 dist-test-slave-rpg6 krb5kdc[2113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:32.486435 2229 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:35:32.488371 2229 webserver.cc:460] Webserver started at http://127.0.97.1:41315/ using document root <none> and password file <none>
I20260620 02:35:32.489251 2229 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:32.489411 2229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:32.489799 2229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:32.493330 2229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "6f15cd6abcd24223822700ff593fae3f"
format_stamp: "Formatted at 2026-06-20 02:35:32 on dist-test-slave-rpg6"
server_key: "53b45c732ac924c881755bc52a035d27"
server_key_iv: "3abeda797ec1d112b86a8ba104227ee6"
server_key_version: "encryptionkey@0"
I20260620 02:35:32.494444 2229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6f15cd6abcd24223822700ff593fae3f"
format_stamp: "Formatted at 2026-06-20 02:35:32 on dist-test-slave-rpg6"
server_key: "53b45c732ac924c881755bc52a035d27"
server_key_iv: "3abeda797ec1d112b86a8ba104227ee6"
server_key_version: "encryptionkey@0"
I20260620 02:35:32.500576 2229 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:35:32.504652 2244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:32.506125 2229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260620 02:35:32.506330 2229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "6f15cd6abcd24223822700ff593fae3f"
format_stamp: "Formatted at 2026-06-20 02:35:32 on dist-test-slave-rpg6"
server_key: "53b45c732ac924c881755bc52a035d27"
server_key_iv: "3abeda797ec1d112b86a8ba104227ee6"
server_key_version: "encryptionkey@0"
I20260620 02:35:32.506557 2229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:32.523403 2229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:32.528703 2229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:32.529047 2229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:32.532169 2229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:32.532330 2229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:32.532481 2229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:32.532581 2229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:35:32 dist-test-slave-rpg6 krb5kdc[2113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:32.558537 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:32.533081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52599 (local address 127.0.97.62:33311)
0620 02:35:32.533586 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:32.533593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:32.534991 (+ 1398us) server_negotiation.cc:407] Connection header received
0620 02:35:32.537548 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:32.537554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:32.537658 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:32.537827 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:32.540621 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:32.541386 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:32.544351 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:32.545158 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:32.545325 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:32.550044 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:32.550056 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:32.550064 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:32.550125 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:32.554683 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:32.555639 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:32.555648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:32.555655 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:32.555824 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:32.556474 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:32.556482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:32.556487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:32.556902 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:32.557070 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:32.557917 (+ 847us) server_negotiation.cc:299] Negotiation successful
0620 02:35:32.558164 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":249,"threads_started":1}
I20260620 02:35:32.559646 2254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:32.533802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33311 (local address 127.0.97.1:52599)
0620 02:35:32.534764 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:32.534826 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:32.537187 (+ 2361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:32.538101 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:32.538110 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:32.539125 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:32.540357 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:32.540398 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:32.541592 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:32.541601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:32.544096 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:32.544114 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:32.545492 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:32.545502 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:32.545723 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:32.546573 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:32.546623 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:32.549662 (+ 3039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:32.554936 (+ 5274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:32.554945 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:32.554961 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:32.555483 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:32.555986 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:32.555992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:32.555996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:32.556321 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:32.557051 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:32.557059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:32.557503 (+ 444us) client_negotiation.cc:769] Sending connection context
0620 02:35:32.557805 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 02:35:32.558392 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":411,"threads_started":1}
I20260620 02:35:32.584440 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:32.567438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47081 (local address 127.0.97.62:33311)
0620 02:35:32.567724 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:32.567735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:32.568127 (+ 392us) server_negotiation.cc:407] Connection header received
0620 02:35:32.568491 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:32.568495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:32.568557 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:32.568668 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:32.570113 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:32.570812 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:32.572971 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:32.573466 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:32.573599 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:32.576452 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:32.576462 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:32.576469 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:32.576510 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:32.580615 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:32.581852 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:32.581862 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:32.581869 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:32.582074 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:32.582617 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:32.582624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:32.582627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:32.583034 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:32.583195 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:32.583713 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 02:35:32.584025 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 02:35:32.584689 2303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:32.566733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33311 (local address 127.0.97.1:47081)
0620 02:35:32.567990 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:32.568014 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:32.568202 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:32.568691 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:32.568699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:32.569437 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:32.569942 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:32.569957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:32.571004 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:32.571007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:32.572761 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:32.572817 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:32.573685 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:32.573695 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:32.573928 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:32.574629 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:32.574663 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:32.576135 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:32.580842 (+ 4707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:32.580852 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:32.580857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:32.581649 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:32.582226 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:32.582233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:32.582239 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:32.582444 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:32.583173 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:32.583177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:32.583372 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 02:35:32.583595 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 02:35:32.584069 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"thread_start_us":912,"threads_started":1}
I20260620 02:35:32.596398 2229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35153
I20260620 02:35:32.596419 2364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35153 every 8 connection(s)
I20260620 02:35:32.597745 2229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:35:32.600351 2365 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33311
I20260620 02:35:32.600607 2365 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:32.601344 2365 heartbeater.cc:511] Master 127.0.97.62:33311 requested a full tablet report, sending...
I20260620 02:35:32.603252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 2229
I20260620 02:35:32.603453 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:35:32.603626 2158 ts_manager.cc:194] Registered new tserver with Master: 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153)
I20260620 02:35:32.604275 388 external_mini_cluster.cc:1428] Setting key 799e765900e30ee2ab5f71ef0029770d
I20260620 02:35:32.605413 2158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52599
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:35:32.651643 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33311
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:33.005777 2369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:33.006179 2369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:33.006371 2369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:33.014369 2369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:33.014554 2369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:33.014750 2369 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:35:33.014847 2369 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:35:33.014986 2369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:35:33.025470 2369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:33.026892 2369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:33.028368 2369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:33.042893 2375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:33.042934 2374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:33.042838 2377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:33.043861 2369 server_base.cc:1034] running on GCE node
I20260620 02:35:33.044545 2369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:33.045495 2369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:33.046749 2369 hybrid_clock.cc:648] HybridClock initialized: now 1781922933046692 us; error 72 us; skew 500 ppm
I20260620 02:35:33.047120 2369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:33 dist-test-slave-rpg6 krb5kdc[2113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:33.050895 2369 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:35:33.052806 2369 webserver.cc:460] Webserver started at http://127.0.97.2:41589/ using document root <none> and password file <none>
I20260620 02:35:33.053620 2369 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:33.053791 2369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:33.054157 2369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:33.057611 2369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f"
format_stamp: "Formatted at 2026-06-20 02:35:33 on dist-test-slave-rpg6"
server_key: "35b7e63936d8fbbcc537717d19922c38"
server_key_iv: "8e9edbdf55ee8a659f2c302d4533b62b"
server_key_version: "encryptionkey@0"
I20260620 02:35:33.058554 2369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f"
format_stamp: "Formatted at 2026-06-20 02:35:33 on dist-test-slave-rpg6"
server_key: "35b7e63936d8fbbcc537717d19922c38"
server_key_iv: "8e9edbdf55ee8a659f2c302d4533b62b"
server_key_version: "encryptionkey@0"
I20260620 02:35:33.064796 2369 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 02:35:33.069347 2384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:33.071249 2369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:35:33.071451 2369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f"
format_stamp: "Formatted at 2026-06-20 02:35:33 on dist-test-slave-rpg6"
server_key: "35b7e63936d8fbbcc537717d19922c38"
server_key_iv: "8e9edbdf55ee8a659f2c302d4533b62b"
server_key_version: "encryptionkey@0"
I20260620 02:35:33.071681 2369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:33.100977 2369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:33.106176 2369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:33.106513 2369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:33.109449 2369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:33.109616 2369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:33.109750 2369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:33.109871 2369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:35:33 dist-test-slave-rpg6 krb5kdc[2113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:33.137262 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.110043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58049 (local address 127.0.97.62:33311)
0620 02:35:33.110853 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:33.110861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:33.112193 (+ 1332us) server_negotiation.cc:407] Connection header received
0620 02:35:33.114324 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:33.114330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:33.114426 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:33.114561 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:33.117215 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.118366 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.121454 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.122313 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.122493 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.128932 (+ 6439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:33.128942 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:33.128949 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:33.129031 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:33.132585 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:33.133893 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:33.133897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:33.133900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:33.134023 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:33.134654 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:33.134659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:33.134665 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:33.135077 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:33.135261 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:33.136468 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0620 02:35:33.136750 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":568,"threads_started":1}
I20260620 02:35:33.139283 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.110870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33311 (local address 127.0.97.2:58049)
0620 02:35:33.111972 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:33.112029 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:33.113994 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:33.114888 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:33.114905 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:33.115784 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:33.116981 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.117016 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.118569 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.118574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.121220 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.121241 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.122644 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.122658 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.123087 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.124188 (+ 1101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:33.124237 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:33.128553 (+ 4316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:33.132784 (+ 4231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:33.132799 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:33.132840 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:33.133721 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:33.134147 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:33.134155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:33.134162 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:33.134508 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:33.135245 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:33.135256 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:33.135915 (+ 659us) client_negotiation.cc:769] Sending connection context
0620 02:35:33.136348 (+ 433us) client_negotiation.cc:240] Negotiation successful
0620 02:35:33.137117 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":404,"threads_started":1}
I20260620 02:35:33.152240 2369 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44961
I20260620 02:35:33.152328 2504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44961 every 8 connection(s)
I20260620 02:35:33.153812 2369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:35:33.156502 2505 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33311
I20260620 02:35:33.156875 2505 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:33.157617 2505 heartbeater.cc:511] Master 127.0.97.62:33311 requested a full tablet report, sending...
I20260620 02:35:33.159720 2158 ts_manager.cc:194] Registered new tserver with Master: f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961)
I20260620 02:35:33.160948 2158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58049
I20260620 02:35:33.162881 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 2369
I20260620 02:35:33.163079 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:35:33.163585 388 external_mini_cluster.cc:1428] Setting key 1f9dcc131cf2d196ef1d5b5733b80612
I20260620 02:35:33.166152 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.149326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45197 (local address 127.0.97.62:33311)
0620 02:35:33.149570 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:33.149577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:33.150389 (+ 812us) server_negotiation.cc:407] Connection header received
0620 02:35:33.150489 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:33.150493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:33.150560 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:33.150646 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:33.152395 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.153335 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.155690 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.156439 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.156575 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.158858 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:33.158865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:33.158870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:33.158908 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:33.162905 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:33.163900 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:33.163907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:33.163910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:33.164090 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:33.164545 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:33.164554 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:33.164558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:33.164976 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:33.165095 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:33.165698 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 02:35:33.165925 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:35:33.166379 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.149319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33311 (local address 127.0.97.2:45197)
0620 02:35:33.150154 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:33.150183 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:33.150362 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:33.150727 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:33.150734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:33.151590 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:33.152179 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.152198 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.153492 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.153503 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.155483 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.155500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.156682 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.156689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.156889 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.157776 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:33.157800 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:33.158682 (+ 882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:33.163097 (+ 4415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:33.163116 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:33.163122 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:33.163676 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:33.164242 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:33.164247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:33.164250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:33.164405 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:33.165141 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:33.165145 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:33.165358 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 02:35:33.165525 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 02:35:33.165866 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":493,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:35:33.212216 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33311
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:33.555702 2509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:33.556099 2509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:33.556272 2509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:33.563896 2509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:33.564091 2509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:33.564254 2509 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:35:33.564342 2509 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:35:33.564447 2509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:35:33.575057 2509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:33.576735 2509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:33.578255 2509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:33.593112 2517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:33.592932 2515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:33.593142 2514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:33.601996 2509 server_base.cc:1034] running on GCE node
I20260620 02:35:33.602708 2509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:33.603757 2509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:33.605044 2509 hybrid_clock.cc:648] HybridClock initialized: now 1781922933604996 us; error 40 us; skew 500 ppm
I20260620 02:35:33.605456 2509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:33 dist-test-slave-rpg6 krb5kdc[2113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:33.608683 2365 heartbeater.cc:503] Master 127.0.97.62:33311 was elected leader, sending a full tablet report...
I20260620 02:35:33.609257 2509 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:35:33.611227 2509 webserver.cc:460] Webserver started at http://127.0.97.3:38295/ using document root <none> and password file <none>
I20260620 02:35:33.612057 2509 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:33.612210 2509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:33.612530 2509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:33.615571 2509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "1c22e6b6a7aa4372aed658cc2672c58f"
format_stamp: "Formatted at 2026-06-20 02:35:33 on dist-test-slave-rpg6"
server_key: "c44f8151cea84e67cfe8a1b547db401e"
server_key_iv: "5a7b4567f2f2043b420f6030a3aadde6"
server_key_version: "encryptionkey@0"
I20260620 02:35:33.616494 2509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1c22e6b6a7aa4372aed658cc2672c58f"
format_stamp: "Formatted at 2026-06-20 02:35:33 on dist-test-slave-rpg6"
server_key: "c44f8151cea84e67cfe8a1b547db401e"
server_key_iv: "5a7b4567f2f2043b420f6030a3aadde6"
server_key_version: "encryptionkey@0"
I20260620 02:35:33.622157 2509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:35:33.626039 2524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:33.627624 2509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:35:33.627913 2509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "1c22e6b6a7aa4372aed658cc2672c58f"
format_stamp: "Formatted at 2026-06-20 02:35:33 on dist-test-slave-rpg6"
server_key: "c44f8151cea84e67cfe8a1b547db401e"
server_key_iv: "5a7b4567f2f2043b420f6030a3aadde6"
server_key_version: "encryptionkey@0"
I20260620 02:35:33.628170 2509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:33.644459 2509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:33.649492 2509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:33.649865 2509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:33.653225 2509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:33.653385 2509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:33.653594 2509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:33.653735 2509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:35:33 dist-test-slave-rpg6 krb5kdc[2113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:33.680595 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.653822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60001 (local address 127.0.97.62:33311)
0620 02:35:33.654040 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:33.654049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:33.655607 (+ 1558us) server_negotiation.cc:407] Connection header received
0620 02:35:33.657866 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:33.657872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:33.657969 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:33.658087 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:33.660469 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.661872 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.664978 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.665654 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.665849 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.671786 (+ 5937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:33.671796 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:33.671804 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:33.671900 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:33.675680 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:33.677219 (+ 1539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:33.677228 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:33.677235 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:33.677452 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:33.677986 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:33.677994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:33.677999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:33.678488 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:33.678704 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:33.679676 (+ 972us) server_negotiation.cc:299] Negotiation successful
0620 02:35:33.680045 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:35:33.681685 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.654486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33311 (local address 127.0.97.3:60001)
0620 02:35:33.655417 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:33.655462 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:33.657542 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:33.658347 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:33.658356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:33.659138 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:33.660153 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.660181 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.662106 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.662118 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.664730 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.664762 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.666063 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.666072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.666441 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.667566 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:33.667613 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:33.671350 (+ 3737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:33.675974 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:33.675990 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:33.676017 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:33.676979 (+ 962us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:33.677587 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:33.677592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:33.677595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:33.677841 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:33.678702 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:33.678710 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:33.679217 (+ 507us) client_negotiation.cc:769] Sending connection context
0620 02:35:33.679497 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 02:35:33.680056 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":424,"threads_started":1}
I20260620 02:35:33.709048 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.690070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60793 (local address 127.0.97.62:33311)
0620 02:35:33.690336 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:33.690347 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:33.690974 (+ 627us) server_negotiation.cc:407] Connection header received
0620 02:35:33.691129 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:33.691138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:33.691262 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:33.691406 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:33.692776 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.694189 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.696979 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.697583 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.697744 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.700703 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:33.700710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:33.700717 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:33.700764 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:33.704870 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:33.706404 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:33.706413 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:33.706418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:33.706594 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:33.707206 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:33.707210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:33.707213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:33.707518 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:33.707651 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:33.708505 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 02:35:33.708781 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:35:33.709823 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.690089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33311 (local address 127.0.97.3:60793)
0620 02:35:33.690823 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:33.690851 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:33.691005 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:33.691455 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:33.691458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:33.692026 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:33.692553 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.692574 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.694464 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.694476 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.696759 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.696781 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.697858 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.697869 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.698124 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.699098 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:33.699137 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:33.700408 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:33.705182 (+ 4774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:33.705196 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:33.705204 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:33.706193 (+ 989us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:33.706770 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:33.706779 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:33.706786 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:33.707002 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:33.707706 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:33.707716 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:33.708081 (+ 365us) client_negotiation.cc:769] Sending connection context
0620 02:35:33.708317 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 02:35:33.708874 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":424,"threads_started":1}
I20260620 02:35:33.716287 2509 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34575
I20260620 02:35:33.716326 2643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34575 every 8 connection(s)
I20260620 02:35:33.717700 2509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:35:33.720109 2644 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33311
I20260620 02:35:33.720371 2644 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:33.721082 2644 heartbeater.cc:511] Master 127.0.97.62:33311 requested a full tablet report, sending...
I20260620 02:35:33.722649 2158 ts_manager.cc:194] Registered new tserver with Master: 1c22e6b6a7aa4372aed658cc2672c58f (127.0.97.3:34575)
I20260620 02:35:33.723249 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 2509
I20260620 02:35:33.723439 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:35:33.723650 2158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60001
I20260620 02:35:33.724072 388 external_mini_cluster.cc:1428] Setting key ee65ab7be482644de5c28b9f6df16a34
I20260620 02:35:33.727365 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:35:33 dist-test-slave-rpg6 krb5kdc[2113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:35:33 dist-test-slave-rpg6 krb5kdc[2113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:33.778456 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.762911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.0.97.62:33311)
0620 02:35:33.763158 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:33.763166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:33.763522 (+ 356us) server_negotiation.cc:407] Connection header received
0620 02:35:33.763743 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:33.763753 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:33.763818 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:33.763986 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:33.765551 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.766373 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.767691 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.768199 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.768342 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.772670 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:33.772675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:33.772679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:33.772718 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:33.775608 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:33.776402 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:33.776410 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:33.776415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:33.776577 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:33.777005 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:33.777011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:33.777014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:33.777316 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:33.777590 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:33.777876 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 02:35:33.778109 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:35:33.782980 2158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:35:33.790241 2158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:35:33.814515 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.804232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44961 (local address 127.0.0.1:43564)
0620 02:35:33.805528 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:33.805599 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:33.805848 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:33.806807 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:33.806816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:33.806862 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:33.807296 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.807308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.809230 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.809238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.811527 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.811541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.813511 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.813517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.813735 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.813789 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 02:35:33.813931 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 02:35:33.814149 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"thread_start_us":1550,"threads_started":1}
I20260620 02:35:33.814518 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.804262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.0.97.2:44961)
0620 02:35:33.804898 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:33.804907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:33.806091 (+ 1184us) server_negotiation.cc:407] Connection header received
0620 02:35:33.806233 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:33.806246 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:33.806522 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:33.806733 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:33.807486 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.809033 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.811735 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.812995 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.813078 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.813323 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:33.814013 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 02:35:33.814207 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":323,"threads_started":1}
I20260620 02:35:33.815596 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.804354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34575 (local address 127.0.0.1:42614)
0620 02:35:33.805982 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:33.806001 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:33.806125 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:33.808156 (+ 2031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:33.808161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:33.808189 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:33.808662 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.808674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.810965 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.810971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.813086 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.813100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.814822 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.814828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.814959 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.815021 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 02:35:33.815093 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 02:35:33.815246 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1480,"thread_start_us":734,"threads_started":1}
I20260620 02:35:33.816092 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.806522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.0.97.1:35153)
0620 02:35:33.807353 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:33.807362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:33.807389 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:35:33.807478 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:33.807486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:33.807748 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:33.808785 (+ 1037us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:33.808988 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.809935 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.813218 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.814709 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.814833 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.815036 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:33.815585 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 02:35:33.815777 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":566,"threads_started":1}
I20260620 02:35:33.816653 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.806178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35153 (local address 127.0.0.1:49736)
0620 02:35:33.806811 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:33.806827 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:33.806931 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:33.807997 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:33.808002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:33.808030 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:33.808536 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.808553 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.810088 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.810097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.813025 (+ 2928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.813039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.815174 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.815179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.815369 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.815411 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 02:35:33.815505 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 02:35:33.815643 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":1,"thread_start_us":376,"threads_started":1}
I20260620 02:35:33.816776 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.805024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42614 (local address 127.0.97.3:34575)
0620 02:35:33.807547 (+ 2523us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:33.807559 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:33.807591 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:35:33.807693 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:33.807699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:33.807945 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:33.808844 (+ 899us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:33.809179 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.810822 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.813355 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.814585 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.815881 (+ 1296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.816119 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:33.816273 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 02:35:33.816442 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2311,"thread_start_us":2228,"threads_started":1}
I20260620 02:35:33.819307 2435 tablet_service.cc:1467] Processing CreateTablet for tablet b86530509a0544d5ae83de318ef152a2 (DEFAULT_TABLE table=test-table [id=e5287d3e943a4fe58d2307ded7fcf8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:33.820132 2574 tablet_service.cc:1467] Processing CreateTablet for tablet b86530509a0544d5ae83de318ef152a2 (DEFAULT_TABLE table=test-table [id=e5287d3e943a4fe58d2307ded7fcf8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:33.820389 2299 tablet_service.cc:1467] Processing CreateTablet for tablet b86530509a0544d5ae83de318ef152a2 (DEFAULT_TABLE table=test-table [id=e5287d3e943a4fe58d2307ded7fcf8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:33.820928 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b86530509a0544d5ae83de318ef152a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:33.821455 2574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b86530509a0544d5ae83de318ef152a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:33.821687 2299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b86530509a0544d5ae83de318ef152a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:33.833338 2663 tablet_bootstrap.cc:492] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: Bootstrap starting.
I20260620 02:35:33.836860 2663 tablet_bootstrap.cc:654] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:33.837908 2663 log.cc:824] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:33.839998 2665 tablet_bootstrap.cc:492] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: Bootstrap starting.
I20260620 02:35:33.840972 2663 tablet_bootstrap.cc:492] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: No bootstrap required, opened a new log
I20260620 02:35:33.841321 2663 ts_tablet_manager.cc:1397] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 02:35:33.844765 2665 tablet_bootstrap.cc:654] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:33.845664 2664 tablet_bootstrap.cc:492] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f: Bootstrap starting.
I20260620 02:35:33.849020 2664 tablet_bootstrap.cc:654] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:33.849313 2665 log.cc:824] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:33.849632 2663 raft_consensus.cc:348] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } }
I20260620 02:35:33.850265 2664 log.cc:824] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:33.850283 2663 raft_consensus.cc:374] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:33.850404 2663 raft_consensus.cc:729] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c22e6b6a7aa4372aed658cc2672c58f, State: Initialized, Role: FOLLOWER
I20260620 02:35:33.851121 2663 consensus_queue.cc:260] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } }
I20260620 02:35:33.852464 2644 heartbeater.cc:503] Master 127.0.97.62:33311 was elected leader, sending a full tablet report...
I20260620 02:35:33.852870 2665 tablet_bootstrap.cc:492] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: No bootstrap required, opened a new log
I20260620 02:35:33.853104 2663 ts_tablet_manager.cc:1428] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 02:35:33.853197 2665 ts_tablet_manager.cc:1397] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260620 02:35:33.856945 2664 tablet_bootstrap.cc:492] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f: No bootstrap required, opened a new log
I20260620 02:35:33.857261 2664 ts_tablet_manager.cc:1397] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20260620 02:35:33.864184 2665 raft_consensus.cc:348] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } }
I20260620 02:35:33.864704 2665 raft_consensus.cc:374] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:33.864807 2665 raft_consensus.cc:729] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f211034c38794d6bbaf0b7f6e9a5ed0f, State: Initialized, Role: FOLLOWER
I20260620 02:35:33.865507 2665 consensus_queue.cc:260] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } }
I20260620 02:35:33.866739 2505 heartbeater.cc:503] Master 127.0.97.62:33311 was elected leader, sending a full tablet report...
I20260620 02:35:33.867502 2665 ts_tablet_manager.cc:1428] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 02:35:33.868191 2664 raft_consensus.cc:348] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } }
I20260620 02:35:33.868709 2664 raft_consensus.cc:374] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:33.868816 2664 raft_consensus.cc:729] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f15cd6abcd24223822700ff593fae3f, State: Initialized, Role: FOLLOWER
I20260620 02:35:33.869576 2664 consensus_queue.cc:260] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } }
I20260620 02:35:33.871459 2664 ts_tablet_manager.cc:1428] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
W20260620 02:35:33.906445 2506 tablet.cc:2369] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:35:33.940214 2671 raft_consensus.cc:482] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:35:33.940567 2671 raft_consensus.cc:504] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } }
I20260620 02:35:33.942286 2671 leader_election.cc:288] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961), 1c22e6b6a7aa4372aed658cc2672c58f (127.0.97.3:34575)
I20260620 02:35:33.949476 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.942742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48793 (local address 127.0.97.2:44961)
0620 02:35:33.942957 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:33.942967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:33.943930 (+ 963us) server_negotiation.cc:407] Connection header received
0620 02:35:33.944169 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:33.944174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:33.944233 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:33.944344 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:33.945082 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.945802 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.947594 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.948429 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.948589 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.948715 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:33.949051 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 02:35:33.949185 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:35:33.949468 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.942829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59759 (local address 127.0.97.3:34575)
0620 02:35:33.943071 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:33.943080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:33.943794 (+ 714us) server_negotiation.cc:407] Connection header received
0620 02:35:33.944112 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:33.944117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:33.944206 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:33.944308 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:33.945016 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.945721 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.947597 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.948423 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.948591 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.948708 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:33.949046 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 02:35:33.949185 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:35:33.949651 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.942712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44961 (local address 127.0.97.1:48793)
0620 02:35:33.943777 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:33.943792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:33.944010 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:33.944372 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:33.944376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:33.944401 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:33.944923 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.944932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.945945 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.945955 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.947427 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.947439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.948715 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.948722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.948883 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.948916 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:35:33.949027 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 02:35:33.949194 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":48,"thread_start_us":564,"threads_started":1}
I20260620 02:35:33.949687 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:33.942563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34575 (local address 127.0.97.1:59759)
0620 02:35:33.943672 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:33.943689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:33.943895 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:33.944346 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:33.944350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:33.944374 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:33.944854 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.944864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.945853 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.945861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.947429 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:33.947441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:33.948668 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:33.948675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:33.948843 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:33.948908 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 02:35:33.949021 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 02:35:33.949206 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":59,"thread_start_us":874,"threads_started":1}
I20260620 02:35:33.950914 2599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b86530509a0544d5ae83de318ef152a2" candidate_uuid: "6f15cd6abcd24223822700ff593fae3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" is_pre_election: true
I20260620 02:35:33.950837 2455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b86530509a0544d5ae83de318ef152a2" candidate_uuid: "6f15cd6abcd24223822700ff593fae3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" is_pre_election: true
I20260620 02:35:33.951573 2455 raft_consensus.cc:2454] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f15cd6abcd24223822700ff593fae3f in term 0.
I20260620 02:35:33.951588 2599 raft_consensus.cc:2454] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f15cd6abcd24223822700ff593fae3f in term 0.
I20260620 02:35:33.952471 2247 leader_election.cc:302] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f15cd6abcd24223822700ff593fae3f, f211034c38794d6bbaf0b7f6e9a5ed0f; no voters:
I20260620 02:35:33.952993 2671 raft_consensus.cc:2781] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:35:33.953164 2671 raft_consensus.cc:482] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:35:33.953281 2671 raft_consensus.cc:3037] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:33.957214 2671 raft_consensus.cc:504] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } }
I20260620 02:35:33.958087 2671 leader_election.cc:288] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [CANDIDATE]: Term 1 election: Requested vote from peers f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961), 1c22e6b6a7aa4372aed658cc2672c58f (127.0.97.3:34575)
I20260620 02:35:33.958804 2599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b86530509a0544d5ae83de318ef152a2" candidate_uuid: "6f15cd6abcd24223822700ff593fae3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c22e6b6a7aa4372aed658cc2672c58f"
I20260620 02:35:33.959074 2599 raft_consensus.cc:3037] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:33.960587 2455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b86530509a0544d5ae83de318ef152a2" candidate_uuid: "6f15cd6abcd24223822700ff593fae3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f"
I20260620 02:35:33.960829 2455 raft_consensus.cc:3037] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:33.963404 2455 raft_consensus.cc:2454] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f15cd6abcd24223822700ff593fae3f in term 1.
I20260620 02:35:33.964102 2247 leader_election.cc:302] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f15cd6abcd24223822700ff593fae3f, f211034c38794d6bbaf0b7f6e9a5ed0f; no voters:
I20260620 02:35:33.964475 2671 raft_consensus.cc:2781] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:33.964989 2671 raft_consensus.cc:686] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 1 LEADER]: Becoming Leader. State: Replica: 6f15cd6abcd24223822700ff593fae3f, State: Running, Role: LEADER
I20260620 02:35:33.965626 2671 consensus_queue.cc:237] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } }
I20260620 02:35:33.966696 2599 raft_consensus.cc:2454] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f15cd6abcd24223822700ff593fae3f in term 1.
W20260620 02:35:33.969513 2645 tablet.cc:2369] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:35:33.970865 2157 catalog_manager.cc:5495] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f15cd6abcd24223822700ff593fae3f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6f15cd6abcd24223822700ff593fae3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } health_report { overall_health: HEALTHY } } }
Jun 20 02:35:34 dist-test-slave-rpg6 krb5kdc[2113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:34.046427 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.030785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.0.97.62:33311)
0620 02:35:34.030967 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:34.030973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:34.031528 (+ 555us) server_negotiation.cc:407] Connection header received
0620 02:35:34.031765 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:34.031768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:34.031884 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:34.031996 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:34.033248 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.034107 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.035336 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.035751 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.035929 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.040556 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:34.040562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:34.040584 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:34.040620 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:34.043532 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:34.044362 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:34.044369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:34.044375 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:34.044500 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:34.044873 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:34.044881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:34.044884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:34.045156 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:34.045325 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:34.045786 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 02:35:34.046121 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":75}
I20260620 02:35:34.046463 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.030803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33311 (local address 127.0.0.1:57532)
0620 02:35:34.031416 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:34.031431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:34.031611 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:34.032025 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:34.032029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:34.032831 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:34.033101 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.033110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.034242 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.034248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.035189 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.035199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.035944 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.035950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.036119 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.037053 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:34.037078 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:34.040336 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:34.043694 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:34.043710 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:34.043719 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:34.044233 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:34.044613 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:34.044619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:34.044622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:34.044770 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:34.045300 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:34.045309 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:34.045479 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 02:35:34.045642 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 02:35:34.045981 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":365,"threads_started":1}
I20260620 02:35:34.052388 2157 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57532:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:35:34.053454 2157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:35:34.061342 2435 tablet_service.cc:1467] Processing CreateTablet for tablet 2047a379b40a49b284cf82c1689900ce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=158a42c531ea43bfa61339aec44d2621]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:35:34.061342 2574 tablet_service.cc:1467] Processing CreateTablet for tablet 2047a379b40a49b284cf82c1689900ce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=158a42c531ea43bfa61339aec44d2621]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:35:34.062428 2574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2047a379b40a49b284cf82c1689900ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:34.062428 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2047a379b40a49b284cf82c1689900ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:34.062318 2299 tablet_service.cc:1467] Processing CreateTablet for tablet 2047a379b40a49b284cf82c1689900ce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=158a42c531ea43bfa61339aec44d2621]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:35:34.063189 2299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2047a379b40a49b284cf82c1689900ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:34.069500 2664 tablet_bootstrap.cc:492] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: Bootstrap starting.
I20260620 02:35:34.070056 2663 tablet_bootstrap.cc:492] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: Bootstrap starting.
I20260620 02:35:34.070823 2665 tablet_bootstrap.cc:492] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f: Bootstrap starting.
I20260620 02:35:34.071424 2664 tablet_bootstrap.cc:654] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:34.071689 2663 tablet_bootstrap.cc:654] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:34.072636 2665 tablet_bootstrap.cc:654] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:34.074663 2664 tablet_bootstrap.cc:492] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: No bootstrap required, opened a new log
I20260620 02:35:34.074829 2663 tablet_bootstrap.cc:492] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: No bootstrap required, opened a new log
I20260620 02:35:34.074863 2664 ts_tablet_manager.cc:1397] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260620 02:35:34.074972 2663 ts_tablet_manager.cc:1397] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 02:35:34.075713 2665 tablet_bootstrap.cc:492] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f: No bootstrap required, opened a new log
I20260620 02:35:34.075714 2664 raft_consensus.cc:348] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.075939 2665 ts_tablet_manager.cc:1397] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 02:35:34.076032 2664 raft_consensus.cc:374] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:34.075860 2663 raft_consensus.cc:348] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.076129 2663 raft_consensus.cc:374] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:34.076128 2664 raft_consensus.cc:729] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f15cd6abcd24223822700ff593fae3f, State: Initialized, Role: FOLLOWER
I20260620 02:35:34.076221 2663 raft_consensus.cc:729] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c22e6b6a7aa4372aed658cc2672c58f, State: Initialized, Role: FOLLOWER
I20260620 02:35:34.076382 2664 consensus_queue.cc:260] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.076452 2663 consensus_queue.cc:260] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.077003 2664 ts_tablet_manager.cc:1428] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:35:34.076879 2665 raft_consensus.cc:348] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.077076 2663 ts_tablet_manager.cc:1428] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 02:35:34.077165 2665 raft_consensus.cc:374] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:34.077276 2665 raft_consensus.cc:729] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f211034c38794d6bbaf0b7f6e9a5ed0f, State: Initialized, Role: FOLLOWER
I20260620 02:35:34.077539 2665 consensus_queue.cc:260] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.078017 2670 tablet_replica.cc:439] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } }
I20260620 02:35:34.078065 2669 tablet_replica.cc:439] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } }
I20260620 02:35:34.078442 2670 tablet_replica.cc:442] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:34.078459 2669 tablet_replica.cc:442] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:34.078724 2671 tablet_replica.cc:439] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } }
I20260620 02:35:34.079018 2671 tablet_replica.cc:442] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:34.079538 2665 ts_tablet_manager.cc:1428] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260620 02:35:34.100605 2366 tablet.cc:2369] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:35:34.151394 2669 raft_consensus.cc:482] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:35:34.151643 2669 raft_consensus.cc:504] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.153654 2669 leader_election.cc:288] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153), f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961)
I20260620 02:35:34.160975 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.154123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34581 (local address 127.0.97.1:35153)
0620 02:35:34.154373 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:34.154380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:34.154401 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:35:34.154476 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:34.154481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:34.154572 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:34.154690 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:34.155316 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.156354 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.158940 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.160053 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.160194 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.160297 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:34.160600 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 02:35:34.160736 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 02:35:34.161041 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.154000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35153 (local address 127.0.97.3:34581)
0620 02:35:34.154218 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:34.154240 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:34.154372 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:34.154872 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:34.154877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:34.154909 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:34.155132 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.155144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.156500 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.156508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.158724 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.158741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.160234 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.160240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.160433 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.160459 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 02:35:34.160570 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 02:35:34.160716 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":71}
I20260620 02:35:34.162104 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.154874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40663 (local address 127.0.97.2:44961)
0620 02:35:34.155039 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:34.155047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:34.155208 (+ 161us) server_negotiation.cc:407] Connection header received
0620 02:35:34.155474 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:34.155480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:34.155552 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:34.155650 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:34.156246 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.157170 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.159208 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.160558 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.160679 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.160777 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:34.161675 (+ 898us) server_negotiation.cc:299] Negotiation successful
0620 02:35:34.161812 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:35:34.161959 2320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2047a379b40a49b284cf82c1689900ce" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f15cd6abcd24223822700ff593fae3f" is_pre_election: true
I20260620 02:35:34.162459 2320 raft_consensus.cc:2454] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 0.
I20260620 02:35:34.163008 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.154447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44961 (local address 127.0.97.3:40663)
0620 02:35:34.155117 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:34.155135 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:34.155307 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:34.155702 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:34.155706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:34.155730 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:34.156079 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.156091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.157403 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.157408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.158984 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.158999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.160927 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.160933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.161490 (+ 557us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.161526 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:35:34.162425 (+ 899us) client_negotiation.cc:240] Negotiation successful
0620 02:35:34.162602 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":2,"thread_start_us":608,"threads_started":1}
I20260620 02:35:34.163168 2527 leader_election.cc:302] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c22e6b6a7aa4372aed658cc2672c58f, 6f15cd6abcd24223822700ff593fae3f; no voters:
I20260620 02:35:34.163609 2669 raft_consensus.cc:2781] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:35:34.163719 2669 raft_consensus.cc:482] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:35:34.163793 2669 raft_consensus.cc:3037] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:34.164040 2455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2047a379b40a49b284cf82c1689900ce" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" is_pre_election: true
I20260620 02:35:34.164317 2455 raft_consensus.cc:2454] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 0.
I20260620 02:35:34.166931 2669 raft_consensus.cc:504] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.167637 2669 leader_election.cc:288] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 election: Requested vote from peers 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153), f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961)
I20260620 02:35:34.168347 2320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2047a379b40a49b284cf82c1689900ce" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f15cd6abcd24223822700ff593fae3f"
I20260620 02:35:34.168370 2455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2047a379b40a49b284cf82c1689900ce" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f"
I20260620 02:35:34.168573 2320 raft_consensus.cc:3037] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:34.168591 2455 raft_consensus.cc:3037] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:34.171675 2320 raft_consensus.cc:2454] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 1.
I20260620 02:35:34.171674 2455 raft_consensus.cc:2454] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 1.
I20260620 02:35:34.172216 2527 leader_election.cc:302] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c22e6b6a7aa4372aed658cc2672c58f, f211034c38794d6bbaf0b7f6e9a5ed0f; no voters:
I20260620 02:35:34.172575 2669 raft_consensus.cc:2781] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:34.173554 2669 raft_consensus.cc:686] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 LEADER]: Becoming Leader. State: Replica: 1c22e6b6a7aa4372aed658cc2672c58f, State: Running, Role: LEADER
I20260620 02:35:34.174157 2669 consensus_queue.cc:237] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.179549 2686 tablet_replica.cc:439] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: TxnStatusTablet state changed. Reason: New leader 1c22e6b6a7aa4372aed658cc2672c58f. Latest consensus state: current_term: 1 leader_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } }
I20260620 02:35:34.179958 2686 tablet_replica.cc:442] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:35:34.180754 2688 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:35:34.180534 2156 catalog_manager.cc:5495] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c22e6b6a7aa4372aed658cc2672c58f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } health_report { overall_health: HEALTHY } } }
I20260620 02:35:34.181054 2688 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:35:34.210781 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.203927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34575 (local address 127.0.0.1:42620)
0620 02:35:34.204193 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:34.204216 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:34.204398 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:34.204922 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:34.204928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:34.204941 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:35:34.205148 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.205161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.206285 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.206291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.208005 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.208021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.208758 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.208765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.209007 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.209102 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:35:34.210195 (+ 1093us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:35:34.210217 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 02:35:34.210296 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 02:35:34.210454 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 02:35:34.210829 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.204021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.0.97.3:34575)
0620 02:35:34.204214 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:34.204218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:34.204295 (+ 77us) server_negotiation.cc:407] Connection header received
0620 02:35:34.204642 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:34.204647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:34.204746 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:34.204870 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:34.205284 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.206141 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.208154 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.208591 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.208706 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.209484 (+ 778us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:34.210075 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:34.210272 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:34.210327 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 02:35:34.210400 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:35:34.219043 2320 raft_consensus.cc:1261] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Refusing update from remote peer 1c22e6b6a7aa4372aed658cc2672c58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:34.219177 2455 raft_consensus.cc:1261] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Refusing update from remote peer 1c22e6b6a7aa4372aed658cc2672c58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:34.220086 2686 consensus_queue.cc:1035] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:34.220505 2669 consensus_queue.cc:1035] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:34.233595 2671 tablet_replica.cc:439] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: TxnStatusTablet state changed. Reason: New leader 1c22e6b6a7aa4372aed658cc2672c58f. Latest consensus state: current_term: 1 leader_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } }
I20260620 02:35:34.233952 2671 tablet_replica.cc:442] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:34.235746 2691 mvcc.cc:205] Tried to move back new op lower bound from 7298756338550530048 to 7298756338382528512. Current Snapshot: MvccSnapshot[applied={T|T < 7298756338550530048}]
I20260620 02:35:34.238986 2669 tablet_replica.cc:439] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } }
I20260620 02:35:34.239262 2669 tablet_replica.cc:442] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:35:34.240231 2670 tablet_replica.cc:439] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f: TxnStatusTablet state changed. Reason: New leader 1c22e6b6a7aa4372aed658cc2672c58f. Latest consensus state: current_term: 1 leader_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } }
I20260620 02:35:34.240522 2670 tablet_replica.cc:442] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:34.242902 2686 tablet_replica.cc:439] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } }
I20260620 02:35:34.243176 2686 tablet_replica.cc:442] T 2047a379b40a49b284cf82c1689900ce P 1c22e6b6a7aa4372aed658cc2672c58f: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:35:34.253381 2671 tablet_replica.cc:439] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } }
I20260620 02:35:34.253871 2671 tablet_replica.cc:442] T 2047a379b40a49b284cf82c1689900ce P 6f15cd6abcd24223822700ff593fae3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:34.264624 2670 tablet_replica.cc:439] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } } }
I20260620 02:35:34.264921 2670 tablet_replica.cc:442] T 2047a379b40a49b284cf82c1689900ce P f211034c38794d6bbaf0b7f6e9a5ed0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:35:34.271044 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.263976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.0.97.1:35153)
0620 02:35:34.264149 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:34.264155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:34.264618 (+ 463us) server_negotiation.cc:407] Connection header received
0620 02:35:34.264730 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:34.264738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:34.264862 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:34.265012 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:34.265606 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.266818 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.268474 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.268910 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.269052 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.269538 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:34.270183 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:34.270340 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:34.270453 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 02:35:34.270544 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:35:34.290537 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.283748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59819 (local address 127.0.97.3:34575)
0620 02:35:34.283970 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:34.283978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:34.284515 (+ 537us) server_negotiation.cc:407] Connection header received
0620 02:35:34.284746 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:34.284753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:34.284835 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:34.284948 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:34.285468 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.286365 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.287996 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.288761 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.288931 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.289694 (+ 763us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:34.289842 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:34.289942 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:34.290124 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 02:35:34.290228 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:35:34.290827 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.283681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34575 (local address 127.0.97.1:59819)
0620 02:35:34.284349 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:34.284366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:34.284574 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:34.284959 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:34.284963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:34.284975 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:35:34.285311 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.285320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.286513 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.286519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.287791 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.287809 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.289030 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.289040 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.289269 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.289381 (+ 112us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:35:34.289968 (+ 587us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:35:34.289982 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 02:35:34.290084 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 02:35:34.290290 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":322,"threads_started":1}
I20260620 02:35:34.304440 2455 raft_consensus.cc:1261] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Refusing update from remote peer 6f15cd6abcd24223822700ff593fae3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:34.304878 2599 raft_consensus.cc:1261] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Refusing update from remote peer 6f15cd6abcd24223822700ff593fae3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:34.305197 2674 consensus_queue.cc:1035] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:34.305900 2674 consensus_queue.cc:1035] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:34.320883 2692 mvcc.cc:205] Tried to move back new op lower bound from 7298756338903113728 to 7298756337528160256. Current Snapshot: MvccSnapshot[applied={T|T < 7298756338903113728}]
I20260620 02:35:34.321977 2693 mvcc.cc:205] Tried to move back new op lower bound from 7298756338903113728 to 7298756337528160256. Current Snapshot: MvccSnapshot[applied={T|T < 7298756338903113728}]
I20260620 02:35:34.354028 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.344776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49337 (local address 127.0.97.1:35153)
0620 02:35:34.344949 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:34.344955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:34.348419 (+ 3464us) server_negotiation.cc:407] Connection header received
0620 02:35:34.348712 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:34.348716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:34.348809 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:34.348889 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:34.349473 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.350463 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.352127 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.352599 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.352679 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.353232 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:34.353346 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:34.353451 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:34.353612 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 02:35:34.353714 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:35:34.354151 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.344366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35153 (local address 127.0.97.3:49337)
0620 02:35:34.348296 (+ 3930us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:34.348324 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:34.348548 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:34.348934 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:34.348939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:34.348952 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:35:34.349304 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.349317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.350589 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.350595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.351953 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.351970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.352841 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.352846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.352977 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.353037 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:35:34.353454 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:35:34.353479 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:35:34.353578 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 02:35:34.353730 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3709,"thread_start_us":3205,"threads_started":1}
I20260620 02:35:34.400931 2158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:34.401254 2158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:34.405898 2158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5: Sending DeleteTablet for 3 replicas of tablet b86530509a0544d5ae83de318ef152a2
I20260620 02:35:34.407248 2435 tablet_service.cc:1514] Processing DeleteTablet for tablet b86530509a0544d5ae83de318ef152a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43564
I20260620 02:35:34.407320 2574 tablet_service.cc:1514] Processing DeleteTablet for tablet b86530509a0544d5ae83de318ef152a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42614
I20260620 02:35:34.408717 2299 tablet_service.cc:1514] Processing DeleteTablet for tablet b86530509a0544d5ae83de318ef152a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49736
I20260620 02:35:34.411785 2158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:35:34.412781 2158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:35:34.421471 2711 tablet_replica.cc:330] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: stopping tablet replica
I20260620 02:35:34.422171 2712 tablet_replica.cc:330] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: stopping tablet replica
I20260620 02:35:34.422850 2712 raft_consensus.cc:2229] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:34.423386 2711 raft_consensus.cc:2229] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:34.424410 2712 raft_consensus.cc:2258] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:34.424563 2711 raft_consensus.cc:2258] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:34.426357 2573 tablet_service.cc:1467] Processing CreateTablet for tablet cde8a158a128429281a805d57066073b (DEFAULT_TABLE table=test-table [id=818c7e43dfd64f19b8ffef1ecc2ffbbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:34.426981 2573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde8a158a128429281a805d57066073b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:34.428169 2711 ts_tablet_manager.cc:1905] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:34.429513 2298 tablet_service.cc:1467] Processing CreateTablet for tablet cde8a158a128429281a805d57066073b (DEFAULT_TABLE table=test-table [id=818c7e43dfd64f19b8ffef1ecc2ffbbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:34.429687 2712 ts_tablet_manager.cc:1905] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:34.429023 2435 tablet_service.cc:1467] Processing CreateTablet for tablet cde8a158a128429281a805d57066073b (DEFAULT_TABLE table=test-table [id=818c7e43dfd64f19b8ffef1ecc2ffbbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:34.430152 2298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde8a158a128429281a805d57066073b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:34.430306 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde8a158a128429281a805d57066073b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:34.437278 2664 tablet_bootstrap.cc:492] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f: Bootstrap starting.
I20260620 02:35:34.439239 2664 tablet_bootstrap.cc:654] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:34.441995 2665 tablet_bootstrap.cc:492] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f: Bootstrap starting.
I20260620 02:35:34.442426 2663 tablet_bootstrap.cc:492] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f: Bootstrap starting.
I20260620 02:35:34.444697 2663 tablet_bootstrap.cc:654] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:34.446091 2665 tablet_bootstrap.cc:654] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:34.453661 2663 tablet_bootstrap.cc:492] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f: No bootstrap required, opened a new log
I20260620 02:35:34.453938 2663 ts_tablet_manager.cc:1397] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20260620 02:35:34.454946 2663 raft_consensus.cc:348] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.455238 2663 raft_consensus.cc:374] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:34.455350 2663 raft_consensus.cc:729] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c22e6b6a7aa4372aed658cc2672c58f, State: Initialized, Role: FOLLOWER
I20260620 02:35:34.455691 2663 consensus_queue.cc:260] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.457295 2713 tablet_replica.cc:330] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f: stopping tablet replica
I20260620 02:35:34.457834 2663 ts_tablet_manager.cc:1428] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:35:34.465665 2713 raft_consensus.cc:2229] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:35:34.466378 2713 raft_consensus.cc:2258] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:34.466737 2665 tablet_bootstrap.cc:492] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f: No bootstrap required, opened a new log
I20260620 02:35:34.466998 2665 ts_tablet_manager.cc:1397] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.005s
I20260620 02:35:34.467883 2665 raft_consensus.cc:348] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.468220 2665 raft_consensus.cc:374] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:34.468358 2665 raft_consensus.cc:729] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f211034c38794d6bbaf0b7f6e9a5ed0f, State: Initialized, Role: FOLLOWER
I20260620 02:35:34.468510 2713 ts_tablet_manager.cc:1905] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:34.468817 2665 consensus_queue.cc:260] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.470683 2665 ts_tablet_manager.cc:1428] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 02:35:34.477521 2664 tablet_bootstrap.cc:492] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f: No bootstrap required, opened a new log
I20260620 02:35:34.477728 2664 ts_tablet_manager.cc:1397] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.010s
I20260620 02:35:34.478201 2712 ts_tablet_manager.cc:1918] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:35:34.477555 2711 ts_tablet_manager.cc:1918] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:35:34.478566 2711 log.cc:1196] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/b86530509a0544d5ae83de318ef152a2
I20260620 02:35:34.478590 2664 raft_consensus.cc:348] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.478884 2664 raft_consensus.cc:374] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:34.478969 2664 raft_consensus.cc:729] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f15cd6abcd24223822700ff593fae3f, State: Initialized, Role: FOLLOWER
I20260620 02:35:34.479298 2664 consensus_queue.cc:260] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.479622 2712 log.cc:1196] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/b86530509a0544d5ae83de318ef152a2
I20260620 02:35:34.480285 2712 ts_tablet_manager.cc:1939] T b86530509a0544d5ae83de318ef152a2 P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting consensus metadata
I20260620 02:35:34.479496 2711 ts_tablet_manager.cc:1939] T b86530509a0544d5ae83de318ef152a2 P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting consensus metadata
I20260620 02:35:34.480902 2664 ts_tablet_manager.cc:1428] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 02:35:34.483444 2144 catalog_manager.cc:4841] TS f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961): tablet b86530509a0544d5ae83de318ef152a2 (table test-table [id=e5287d3e943a4fe58d2307ded7fcf8f8]) successfully deleted
I20260620 02:35:34.484601 2142 catalog_manager.cc:4841] TS 1c22e6b6a7aa4372aed658cc2672c58f (127.0.97.3:34575): tablet b86530509a0544d5ae83de318ef152a2 (table test-table [id=e5287d3e943a4fe58d2307ded7fcf8f8]) successfully deleted
I20260620 02:35:34.487329 2713 ts_tablet_manager.cc:1918] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:35:34.487546 2713 log.cc:1196] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/b86530509a0544d5ae83de318ef152a2
I20260620 02:35:34.488252 2713 ts_tablet_manager.cc:1939] T b86530509a0544d5ae83de318ef152a2 P 6f15cd6abcd24223822700ff593fae3f: Deleting consensus metadata
I20260620 02:35:34.490279 2144 catalog_manager.cc:4841] TS 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153): tablet b86530509a0544d5ae83de318ef152a2 (table test-table [id=e5287d3e943a4fe58d2307ded7fcf8f8]) successfully deleted
I20260620 02:35:34.509037 2686 raft_consensus.cc:482] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:35:34.509323 2686 raft_consensus.cc:504] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.510167 2686 leader_election.cc:288] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153), f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961)
I20260620 02:35:34.510761 2320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde8a158a128429281a805d57066073b" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f15cd6abcd24223822700ff593fae3f" is_pre_election: true
I20260620 02:35:34.510848 2455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde8a158a128429281a805d57066073b" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" is_pre_election: true
I20260620 02:35:34.511022 2320 raft_consensus.cc:2454] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 0.
I20260620 02:35:34.511116 2455 raft_consensus.cc:2454] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 0.
I20260620 02:35:34.511620 2527 leader_election.cc:302] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c22e6b6a7aa4372aed658cc2672c58f, f211034c38794d6bbaf0b7f6e9a5ed0f; no voters:
I20260620 02:35:34.512017 2686 raft_consensus.cc:2781] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:35:34.512154 2686 raft_consensus.cc:482] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:35:34.512274 2686 raft_consensus.cc:3037] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:34.515758 2686 raft_consensus.cc:504] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.516559 2686 leader_election.cc:288] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 election: Requested vote from peers 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153), f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961)
I20260620 02:35:34.517169 2455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde8a158a128429281a805d57066073b" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f"
I20260620 02:35:34.517246 2320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde8a158a128429281a805d57066073b" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f15cd6abcd24223822700ff593fae3f"
I20260620 02:35:34.517390 2455 raft_consensus.cc:3037] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:34.517483 2320 raft_consensus.cc:3037] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:34.520612 2455 raft_consensus.cc:2454] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 1.
I20260620 02:35:34.521210 2527 leader_election.cc:302] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c22e6b6a7aa4372aed658cc2672c58f, f211034c38794d6bbaf0b7f6e9a5ed0f; no voters:
I20260620 02:35:34.521492 2320 raft_consensus.cc:2454] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 1.
I20260620 02:35:34.521652 2686 raft_consensus.cc:2781] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:34.522236 2686 raft_consensus.cc:686] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 LEADER]: Becoming Leader. State: Replica: 1c22e6b6a7aa4372aed658cc2672c58f, State: Running, Role: LEADER
I20260620 02:35:34.522553 2686 consensus_queue.cc:237] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:34.525734 2156 catalog_manager.cc:5495] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c22e6b6a7aa4372aed658cc2672c58f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } health_report { overall_health: HEALTHY } } }
I20260620 02:35:34.556387 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.549156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.0.97.3:34575)
0620 02:35:34.549366 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:34.549372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:34.549478 (+ 106us) server_negotiation.cc:407] Connection header received
0620 02:35:34.549679 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:34.549685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:34.549771 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:34.549887 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:34.550474 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.551729 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.553732 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.554454 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.554602 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.555135 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:34.555316 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:34.555419 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:34.555934 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 02:35:34.556021 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:35:34.574756 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.567866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49113 (local address 127.0.97.3:34575)
0620 02:35:34.568033 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:34.568039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:34.568059 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:35:34.568129 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:34.568135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:34.568221 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:34.568319 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:34.568761 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.570094 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.572194 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.572864 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.572999 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.573526 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:34.573722 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:34.573815 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:34.574246 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 02:35:34.574393 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":55}
I20260620 02:35:34.574759 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:34.567593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34575 (local address 127.0.97.3:49113)
0620 02:35:34.567806 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:34.567854 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:34.567983 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:34.568359 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:34.568365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:34.568381 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:35:34.568606 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.568619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.570306 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.570312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.572030 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:34.572045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:34.573049 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:34.573059 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:34.573279 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:34.573342 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:35:34.573843 (+ 501us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:35:34.573854 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:35:34.573955 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 02:35:34.574128 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 02:35:34.586651 2454 raft_consensus.cc:1261] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Refusing update from remote peer 1c22e6b6a7aa4372aed658cc2672c58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:34.587283 2320 raft_consensus.cc:1261] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Refusing update from remote peer 1c22e6b6a7aa4372aed658cc2672c58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:34.588182 2686 consensus_queue.cc:1035] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:34.588449 2669 consensus_queue.cc:1035] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:35:34.636605 2691 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 02:35:36.289279 2156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:36.289574 2156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:36.297026 2156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5: Sending DeleteTablet for 3 replicas of tablet cde8a158a128429281a805d57066073b
I20260620 02:35:36.298116 2299 tablet_service.cc:1514] Processing DeleteTablet for tablet cde8a158a128429281a805d57066073b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49736
I20260620 02:35:36.298199 2435 tablet_service.cc:1514] Processing DeleteTablet for tablet cde8a158a128429281a805d57066073b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43564
I20260620 02:35:36.298467 2573 tablet_service.cc:1514] Processing DeleteTablet for tablet cde8a158a128429281a805d57066073b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42614
I20260620 02:35:36.309381 2743 tablet_replica.cc:330] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f: stopping tablet replica
I20260620 02:35:36.309551 2741 tablet_replica.cc:330] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f: stopping tablet replica
I20260620 02:35:36.309803 2741 raft_consensus.cc:2229] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:36.309907 2743 raft_consensus.cc:2229] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:35:36.310168 2741 raft_consensus.cc:2258] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:36.310940 2743 raft_consensus.cc:2258] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:36.312377 2743 ts_tablet_manager.cc:1905] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:36.312685 2741 ts_tablet_manager.cc:1905] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:36.320930 2741 ts_tablet_manager.cc:1918] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:35:36.321583 2742 tablet_replica.cc:330] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f: stopping tablet replica
I20260620 02:35:36.321839 2742 raft_consensus.cc:2229] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:36.321918 2743 ts_tablet_manager.cc:1918] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:35:36.322088 2743 log.cc:1196] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/cde8a158a128429281a805d57066073b
I20260620 02:35:36.322113 2742 raft_consensus.cc:2258] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:36.322427 2741 log.cc:1196] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/cde8a158a128429281a805d57066073b
I20260620 02:35:36.322670 2743 ts_tablet_manager.cc:1939] T cde8a158a128429281a805d57066073b P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting consensus metadata
I20260620 02:35:36.323309 2742 ts_tablet_manager.cc:1905] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:36.323989 2142 catalog_manager.cc:4841] TS 1c22e6b6a7aa4372aed658cc2672c58f (127.0.97.3:34575): tablet cde8a158a128429281a805d57066073b (table test-table [id=818c7e43dfd64f19b8ffef1ecc2ffbbb]) successfully deleted
I20260620 02:35:36.328366 2741 ts_tablet_manager.cc:1939] T cde8a158a128429281a805d57066073b P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting consensus metadata
I20260620 02:35:36.329526 2144 catalog_manager.cc:4841] TS f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961): tablet cde8a158a128429281a805d57066073b (table test-table [id=818c7e43dfd64f19b8ffef1ecc2ffbbb]) successfully deleted
I20260620 02:35:36.331442 2742 ts_tablet_manager.cc:1918] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:35:36.331616 2742 log.cc:1196] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/cde8a158a128429281a805d57066073b
I20260620 02:35:36.332172 2742 ts_tablet_manager.cc:1939] T cde8a158a128429281a805d57066073b P 6f15cd6abcd24223822700ff593fae3f: Deleting consensus metadata
I20260620 02:35:36.333640 2144 catalog_manager.cc:4841] TS 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153): tablet cde8a158a128429281a805d57066073b (table test-table [id=818c7e43dfd64f19b8ffef1ecc2ffbbb]) successfully deleted
Jun 20 02:35:36 dist-test-slave-rpg6 krb5kdc[2113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922936, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:35:36 dist-test-slave-rpg6 krb5kdc[2113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922936, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:36.363658 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:36.345414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.0.97.62:33311)
0620 02:35:36.346097 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:36.346106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:36.346136 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:35:36.346211 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:36.346215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:36.346285 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:36.346406 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:36.347884 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:36.348700 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:36.349999 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:36.350420 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:36.350563 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:36.358203 (+ 7640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:36.358209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:36.358212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:36.358255 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:36.360916 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:36.361705 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:36.361711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:36.361717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:36.361850 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:36.362322 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:36.362326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:36.362328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:36.362612 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:36.362894 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:36.363116 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 02:35:36.363304 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":354,"threads_started":1}
I20260620 02:35:36.368091 2156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:35:36.368780 2156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:35:36.376596 2299 tablet_service.cc:1467] Processing CreateTablet for tablet c8c7441227ae4b37bb455024465df243 (DEFAULT_TABLE table=test-table [id=40efd3f0f03f4d8e960c3e642fa5847c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:36.376951 2573 tablet_service.cc:1467] Processing CreateTablet for tablet c8c7441227ae4b37bb455024465df243 (DEFAULT_TABLE table=test-table [id=40efd3f0f03f4d8e960c3e642fa5847c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:36.377128 2299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c7441227ae4b37bb455024465df243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:36.377430 2573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c7441227ae4b37bb455024465df243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:36.382301 2752 tablet_bootstrap.cc:492] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f: Bootstrap starting.
I20260620 02:35:36.382304 2435 tablet_service.cc:1467] Processing CreateTablet for tablet c8c7441227ae4b37bb455024465df243 (DEFAULT_TABLE table=test-table [id=40efd3f0f03f4d8e960c3e642fa5847c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:36.382763 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c7441227ae4b37bb455024465df243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:36.382805 2753 tablet_bootstrap.cc:492] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f: Bootstrap starting.
I20260620 02:35:36.384570 2753 tablet_bootstrap.cc:654] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:36.385746 2752 tablet_bootstrap.cc:654] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:36.387341 2753 tablet_bootstrap.cc:492] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f: No bootstrap required, opened a new log
I20260620 02:35:36.387526 2753 ts_tablet_manager.cc:1397] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 02:35:36.388475 2753 raft_consensus.cc:348] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.388710 2753 raft_consensus.cc:374] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:36.388890 2752 tablet_bootstrap.cc:492] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f: No bootstrap required, opened a new log
I20260620 02:35:36.388888 2753 raft_consensus.cc:729] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c22e6b6a7aa4372aed658cc2672c58f, State: Initialized, Role: FOLLOWER
I20260620 02:35:36.389026 2752 ts_tablet_manager.cc:1397] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 02:35:36.389214 2753 consensus_queue.cc:260] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.389926 2753 ts_tablet_manager.cc:1428] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 02:35:36.389846 2752 raft_consensus.cc:348] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.390148 2752 raft_consensus.cc:374] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:36.390435 2752 raft_consensus.cc:729] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f15cd6abcd24223822700ff593fae3f, State: Initialized, Role: FOLLOWER
I20260620 02:35:36.390796 2752 consensus_queue.cc:260] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.395191 2752 ts_tablet_manager.cc:1428] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 02:35:36.395900 2756 tablet_bootstrap.cc:492] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f: Bootstrap starting.
I20260620 02:35:36.398090 2756 tablet_bootstrap.cc:654] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:36.400370 2756 tablet_bootstrap.cc:492] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f: No bootstrap required, opened a new log
I20260620 02:35:36.400512 2756 ts_tablet_manager.cc:1397] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 02:35:36.401203 2756 raft_consensus.cc:348] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.401423 2756 raft_consensus.cc:374] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:36.401490 2756 raft_consensus.cc:729] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f211034c38794d6bbaf0b7f6e9a5ed0f, State: Initialized, Role: FOLLOWER
I20260620 02:35:36.401759 2756 consensus_queue.cc:260] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.402681 2756 ts_tablet_manager.cc:1428] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:35:36.446718 2736 raft_consensus.cc:482] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:35:36.447014 2736 raft_consensus.cc:504] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.447798 2736 leader_election.cc:288] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153), f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961)
I20260620 02:35:36.448447 2319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c7441227ae4b37bb455024465df243" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f15cd6abcd24223822700ff593fae3f" is_pre_election: true
I20260620 02:35:36.448482 2454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c7441227ae4b37bb455024465df243" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" is_pre_election: true
I20260620 02:35:36.448719 2319 raft_consensus.cc:2454] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 0.
I20260620 02:35:36.448817 2454 raft_consensus.cc:2454] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 0.
I20260620 02:35:36.449271 2527 leader_election.cc:302] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c22e6b6a7aa4372aed658cc2672c58f, 6f15cd6abcd24223822700ff593fae3f; no voters:
I20260620 02:35:36.449541 2736 raft_consensus.cc:2781] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:35:36.449658 2736 raft_consensus.cc:482] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:35:36.449756 2736 raft_consensus.cc:3037] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:36.452073 2736 raft_consensus.cc:504] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.452728 2736 leader_election.cc:288] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 election: Requested vote from peers 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153), f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961)
I20260620 02:35:36.453271 2319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c7441227ae4b37bb455024465df243" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f15cd6abcd24223822700ff593fae3f"
I20260620 02:35:36.453322 2454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c7441227ae4b37bb455024465df243" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f"
I20260620 02:35:36.453517 2319 raft_consensus.cc:3037] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:36.453563 2454 raft_consensus.cc:3037] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:36.455608 2454 raft_consensus.cc:2454] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 1.
I20260620 02:35:36.456143 2527 leader_election.cc:302] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c22e6b6a7aa4372aed658cc2672c58f, f211034c38794d6bbaf0b7f6e9a5ed0f; no voters:
I20260620 02:35:36.456411 2319 raft_consensus.cc:2454] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 1.
I20260620 02:35:36.456586 2736 raft_consensus.cc:2781] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:36.456760 2736 raft_consensus.cc:686] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 LEADER]: Becoming Leader. State: Replica: 1c22e6b6a7aa4372aed658cc2672c58f, State: Running, Role: LEADER
I20260620 02:35:36.457121 2736 consensus_queue.cc:237] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.459960 2158 catalog_manager.cc:5495] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c22e6b6a7aa4372aed658cc2672c58f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } health_report { overall_health: HEALTHY } } }
I20260620 02:35:36.508792 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:36.498953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.0.97.3:34575)
0620 02:35:36.499641 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:36.499650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:36.499682 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:35:36.499766 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:36.499772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:36.500030 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:36.500145 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:36.500622 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:36.502350 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:36.505405 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:36.506075 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:36.506278 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:36.507110 (+ 832us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:36.507458 (+ 348us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:36.508189 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:36.508252 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 02:35:36.508376 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":376,"threads_started":1}
I20260620 02:35:36.519354 2319 raft_consensus.cc:1261] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Refusing update from remote peer 1c22e6b6a7aa4372aed658cc2672c58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:36.519397 2454 raft_consensus.cc:1261] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Refusing update from remote peer 1c22e6b6a7aa4372aed658cc2672c58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:36.519992 2736 consensus_queue.cc:1035] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:36.520488 2736 consensus_queue.cc:1035] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:36.570969 2155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:36.571197 2155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:36.574571 2155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5: Sending DeleteTablet for 3 replicas of tablet c8c7441227ae4b37bb455024465df243
I20260620 02:35:36.575551 2299 tablet_service.cc:1514] Processing DeleteTablet for tablet c8c7441227ae4b37bb455024465df243 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49736
I20260620 02:35:36.575680 2573 tablet_service.cc:1514] Processing DeleteTablet for tablet c8c7441227ae4b37bb455024465df243 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42614
I20260620 02:35:36.575987 2743 tablet_replica.cc:330] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f: stopping tablet replica
I20260620 02:35:36.576228 2743 raft_consensus.cc:2229] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:35:36.576601 2743 raft_consensus.cc:2258] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:36.576990 2742 tablet_replica.cc:330] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f: stopping tablet replica
I20260620 02:35:36.576905 2155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 02:35:36.577209 2742 raft_consensus.cc:2229] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:36.577394 2742 raft_consensus.cc:2258] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 02:35:36.577661 2155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:35:36.577749 2435 tablet_service.cc:1514] Processing DeleteTablet for tablet c8c7441227ae4b37bb455024465df243 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43564
I20260620 02:35:36.577880 2743 ts_tablet_manager.cc:1905] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:36.578056 2741 tablet_replica.cc:330] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f: stopping tablet replica
I20260620 02:35:36.578274 2741 raft_consensus.cc:2229] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:36.578469 2741 raft_consensus.cc:2258] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:36.578677 2742 ts_tablet_manager.cc:1905] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:36.579617 2741 ts_tablet_manager.cc:1905] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:36.583217 2742 ts_tablet_manager.cc:1918] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:35:36.583340 2742 log.cc:1196] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/c8c7441227ae4b37bb455024465df243
I20260620 02:35:36.583632 2742 ts_tablet_manager.cc:1939] T c8c7441227ae4b37bb455024465df243 P 6f15cd6abcd24223822700ff593fae3f: Deleting consensus metadata
I20260620 02:35:36.585134 2144 catalog_manager.cc:4841] TS 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153): tablet c8c7441227ae4b37bb455024465df243 (table test-table [id=40efd3f0f03f4d8e960c3e642fa5847c]) successfully deleted
I20260620 02:35:36.586350 2743 ts_tablet_manager.cc:1918] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:35:36.586396 2741 ts_tablet_manager.cc:1918] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:35:36.586534 2743 log.cc:1196] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/c8c7441227ae4b37bb455024465df243
I20260620 02:35:36.586570 2741 log.cc:1196] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/c8c7441227ae4b37bb455024465df243
I20260620 02:35:36.586998 2741 ts_tablet_manager.cc:1939] T c8c7441227ae4b37bb455024465df243 P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting consensus metadata
I20260620 02:35:36.587648 2743 ts_tablet_manager.cc:1939] T c8c7441227ae4b37bb455024465df243 P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting consensus metadata
I20260620 02:35:36.588138 2144 catalog_manager.cc:4841] TS f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961): tablet c8c7441227ae4b37bb455024465df243 (table test-table [id=40efd3f0f03f4d8e960c3e642fa5847c]) successfully deleted
I20260620 02:35:36.588922 2142 catalog_manager.cc:4841] TS 1c22e6b6a7aa4372aed658cc2672c58f (127.0.97.3:34575): tablet c8c7441227ae4b37bb455024465df243 (table test-table [id=40efd3f0f03f4d8e960c3e642fa5847c]) successfully deleted
I20260620 02:35:36.592386 2299 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7084337476432f89b14dbf104bdbdc (DEFAULT_TABLE table=test-table [id=f75eddc922224784a123a873af5d1db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:36.592406 2573 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7084337476432f89b14dbf104bdbdc (DEFAULT_TABLE table=test-table [id=f75eddc922224784a123a873af5d1db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:36.592384 2435 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7084337476432f89b14dbf104bdbdc (DEFAULT_TABLE table=test-table [id=f75eddc922224784a123a873af5d1db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:36.592844 2573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7084337476432f89b14dbf104bdbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:36.592862 2299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7084337476432f89b14dbf104bdbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:36.592911 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7084337476432f89b14dbf104bdbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:36.598378 2756 tablet_bootstrap.cc:492] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f: Bootstrap starting.
I20260620 02:35:36.600106 2756 tablet_bootstrap.cc:654] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:36.600391 2752 tablet_bootstrap.cc:492] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f: Bootstrap starting.
I20260620 02:35:36.601133 2753 tablet_bootstrap.cc:492] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f: Bootstrap starting.
I20260620 02:35:36.602401 2752 tablet_bootstrap.cc:654] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:36.603266 2753 tablet_bootstrap.cc:654] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:36.605288 2752 tablet_bootstrap.cc:492] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f: No bootstrap required, opened a new log
I20260620 02:35:36.605571 2752 ts_tablet_manager.cc:1397] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 02:35:36.606338 2752 raft_consensus.cc:348] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.606582 2752 raft_consensus.cc:374] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:36.606666 2756 tablet_bootstrap.cc:492] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f: No bootstrap required, opened a new log
I20260620 02:35:36.606673 2752 raft_consensus.cc:729] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f15cd6abcd24223822700ff593fae3f, State: Initialized, Role: FOLLOWER
I20260620 02:35:36.606863 2756 ts_tablet_manager.cc:1397] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260620 02:35:36.607129 2752 consensus_queue.cc:260] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.607788 2756 raft_consensus.cc:348] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.608093 2756 raft_consensus.cc:374] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:36.608198 2756 raft_consensus.cc:729] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f211034c38794d6bbaf0b7f6e9a5ed0f, State: Initialized, Role: FOLLOWER
I20260620 02:35:36.608471 2756 consensus_queue.cc:260] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.609387 2753 tablet_bootstrap.cc:492] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f: No bootstrap required, opened a new log
I20260620 02:35:36.609639 2753 ts_tablet_manager.cc:1397] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260620 02:35:36.610625 2752 ts_tablet_manager.cc:1428] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 02:35:36.610481 2753 raft_consensus.cc:348] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.610754 2753 raft_consensus.cc:374] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:36.610850 2753 raft_consensus.cc:729] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c22e6b6a7aa4372aed658cc2672c58f, State: Initialized, Role: FOLLOWER
I20260620 02:35:36.611104 2753 consensus_queue.cc:260] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.611711 2753 ts_tablet_manager.cc:1428] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 02:35:36.613379 2756 ts_tablet_manager.cc:1428] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 02:35:36.836249 2761 raft_consensus.cc:482] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:35:36.836457 2761 raft_consensus.cc:504] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.837339 2761 leader_election.cc:288] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153), f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961)
I20260620 02:35:36.838002 2319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7084337476432f89b14dbf104bdbdc" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f15cd6abcd24223822700ff593fae3f" is_pre_election: true
I20260620 02:35:36.838021 2455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7084337476432f89b14dbf104bdbdc" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" is_pre_election: true
I20260620 02:35:36.838301 2319 raft_consensus.cc:2454] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 0.
I20260620 02:35:36.838317 2455 raft_consensus.cc:2454] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 0.
I20260620 02:35:36.838795 2527 leader_election.cc:302] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c22e6b6a7aa4372aed658cc2672c58f, f211034c38794d6bbaf0b7f6e9a5ed0f; no voters:
I20260620 02:35:36.839169 2761 raft_consensus.cc:2781] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:35:36.839280 2761 raft_consensus.cc:482] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:35:36.839442 2761 raft_consensus.cc:3037] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:36.841650 2761 raft_consensus.cc:504] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.842378 2761 leader_election.cc:288] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 election: Requested vote from peers 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153), f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961)
I20260620 02:35:36.843041 2319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7084337476432f89b14dbf104bdbdc" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f15cd6abcd24223822700ff593fae3f"
I20260620 02:35:36.843267 2319 raft_consensus.cc:3037] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:36.843240 2455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7084337476432f89b14dbf104bdbdc" candidate_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f"
I20260620 02:35:36.843536 2455 raft_consensus.cc:3037] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:36.845615 2319 raft_consensus.cc:2454] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 1.
I20260620 02:35:36.845608 2455 raft_consensus.cc:2454] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c22e6b6a7aa4372aed658cc2672c58f in term 1.
I20260620 02:35:36.846136 2527 leader_election.cc:302] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c22e6b6a7aa4372aed658cc2672c58f, f211034c38794d6bbaf0b7f6e9a5ed0f; no voters:
I20260620 02:35:36.846463 2761 raft_consensus.cc:2781] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:36.846616 2761 raft_consensus.cc:686] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 LEADER]: Becoming Leader. State: Replica: 1c22e6b6a7aa4372aed658cc2672c58f, State: Running, Role: LEADER
I20260620 02:35:36.846933 2761 consensus_queue.cc:237] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } }
I20260620 02:35:36.849702 2157 catalog_manager.cc:5495] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c22e6b6a7aa4372aed658cc2672c58f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c22e6b6a7aa4372aed658cc2672c58f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34575 } health_report { overall_health: HEALTHY } } }
I20260620 02:35:36.933337 2319 raft_consensus.cc:1261] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Refusing update from remote peer 1c22e6b6a7aa4372aed658cc2672c58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:36.933386 2455 raft_consensus.cc:1261] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Refusing update from remote peer 1c22e6b6a7aa4372aed658cc2672c58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:36.933933 2669 consensus_queue.cc:1035] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f15cd6abcd24223822700ff593fae3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:36.934280 2761 consensus_queue.cc:1035] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f211034c38794d6bbaf0b7f6e9a5ed0f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:35:36.958461 2722 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 02:35:38.547793 2155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:38.548064 2155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:38.552354 2155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0611fa41457c464c9ef4a48227f37cd5: Sending DeleteTablet for 3 replicas of tablet 5b7084337476432f89b14dbf104bdbdc
I20260620 02:35:38.553272 2299 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7084337476432f89b14dbf104bdbdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49736
I20260620 02:35:38.553341 2435 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7084337476432f89b14dbf104bdbdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43564
I20260620 02:35:38.553349 2573 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7084337476432f89b14dbf104bdbdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42614
I20260620 02:35:38.555804 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 2229
I20260620 02:35:38.556897 2794 tablet_replica.cc:330] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f: stopping tablet replica
I20260620 02:35:38.557087 2796 tablet_replica.cc:330] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f: stopping tablet replica
I20260620 02:35:38.557117 2794 raft_consensus.cc:2229] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:38.557266 2796 raft_consensus.cc:2229] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:38.557394 2794 raft_consensus.cc:2258] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:38.557483 2796 raft_consensus.cc:2258] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:38.558525 2795 tablet_replica.cc:330] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f: stopping tablet replica
I20260620 02:35:38.559049 2794 ts_tablet_manager.cc:1905] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:38.559265 2796 ts_tablet_manager.cc:1905] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:38.559499 2795 raft_consensus.cc:2229] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:35:38.560043 2795 raft_consensus.cc:2258] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:38.561311 2795 ts_tablet_manager.cc:1905] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:38.566282 2794 ts_tablet_manager.cc:1918] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:35:38.566454 2794 log.cc:1196] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/5b7084337476432f89b14dbf104bdbdc
I20260620 02:35:38.566701 2796 ts_tablet_manager.cc:1918] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:35:38.566869 2796 log.cc:1196] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/5b7084337476432f89b14dbf104bdbdc
I20260620 02:35:38.566959 2794 ts_tablet_manager.cc:1939] T 5b7084337476432f89b14dbf104bdbdc P 6f15cd6abcd24223822700ff593fae3f: Deleting consensus metadata
I20260620 02:35:38.567314 2796 ts_tablet_manager.cc:1939] T 5b7084337476432f89b14dbf104bdbdc P f211034c38794d6bbaf0b7f6e9a5ed0f: Deleting consensus metadata
I20260620 02:35:38.568251 2144 catalog_manager.cc:4841] TS 6f15cd6abcd24223822700ff593fae3f (127.0.97.1:35153): tablet 5b7084337476432f89b14dbf104bdbdc (table test-table [id=f75eddc922224784a123a873af5d1db3]) successfully deleted
I20260620 02:35:38.568583 2144 catalog_manager.cc:4841] TS f211034c38794d6bbaf0b7f6e9a5ed0f (127.0.97.2:44961): tablet 5b7084337476432f89b14dbf104bdbdc (table test-table [id=f75eddc922224784a123a873af5d1db3]) successfully deleted
I20260620 02:35:38.569792 2795 ts_tablet_manager.cc:1918] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:35:38.569952 2795 log.cc:1196] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/5b7084337476432f89b14dbf104bdbdc
I20260620 02:35:38.570569 2795 ts_tablet_manager.cc:1939] T 5b7084337476432f89b14dbf104bdbdc P 1c22e6b6a7aa4372aed658cc2672c58f: Deleting consensus metadata
I20260620 02:35:38.571744 2142 catalog_manager.cc:4841] TS 1c22e6b6a7aa4372aed658cc2672c58f (127.0.97.3:34575): tablet 5b7084337476432f89b14dbf104bdbdc (table test-table [id=f75eddc922224784a123a873af5d1db3]) successfully deleted
Jun 20 02:35:38 dist-test-slave-rpg6 krb5kdc[2113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922936, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:35:38.583206 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:38.557525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.0.97.1:35153)
0620 02:35:38.559459 (+ 1934us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:38.559465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:38.559494 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:35:38.559559 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:38.559564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:38.559654 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:38.560378 (+ 724us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:38.561653 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:38.562992 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:38.565174 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:38.565855 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:38.565993 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:38.576372 (+ 10379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:38.576392 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:38.576410 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:38.576456 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:38.580215 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:38.581034 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:38.581041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:38.581044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:38.581190 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:38.581587 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:38.581596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:38.581598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:38.581973 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:38.582281 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:38.582608 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 02:35:38.582833 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1800,"thread_start_us":1684,"threads_started":1}
W20260620 02:35:38.584019 2360 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49754
W20260620 02:35:38.584760 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 2229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:35:38.584991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 2229
I20260620 02:35:38.608011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 2369
Jun 20 02:35:38 dist-test-slave-rpg6 krb5kdc[2113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922936, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:35:38.624428 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:38.608976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.0.97.2:44961)
0620 02:35:38.609537 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:38.609541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:38.609558 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:35:38.609610 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:38.609614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:38.609670 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:38.609776 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:38.611020 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:38.611769 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:38.613060 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:38.613480 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:38.613578 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:38.617878 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:38.617898 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:38.617915 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:38.617960 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:38.621448 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:38.622301 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:38.622310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:38.622312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:38.622467 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:38.622835 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:38.622839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:38.622841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:38.623282 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:38.623512 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:38.623966 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 02:35:38.624161 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":310,"threads_started":1}
W20260620 02:35:38.625286 2499 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43570
W20260620 02:35:38.626128 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 2369: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:35:38.626276 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 2369
I20260620 02:35:38.644620 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 2509
Jun 20 02:35:38 dist-test-slave-rpg6 krb5kdc[2113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922936, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:35:38.660848 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:38.645519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.0.97.3:34575)
0620 02:35:38.646050 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:38.646055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:38.646071 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:35:38.646124 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:38.646128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:38.646189 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:38.646290 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:38.647701 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:38.648538 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:38.649884 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:38.650378 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:38.650488 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:38.654986 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:38.655001 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:38.655017 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:38.655057 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:38.658191 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:38.658994 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:38.659002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:38.659004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:38.659133 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:38.659455 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:38.659459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:38.659461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:38.659802 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:38.660079 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:38.660382 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 02:35:38.660577 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":298,"threads_started":1}
W20260620 02:35:38.661798 2639 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42644
W20260620 02:35:38.662464 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 2509: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:35:38.662642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 2509
I20260620 02:35:38.688715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 2126
I20260620 02:35:38.689996 2198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:38.809911 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 2126
2026-06-20T02:35:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8700 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:35:38 dist-test-slave-rpg6 krb5kdc[2804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:35:38 dist-test-slave-rpg6 krb5kdc[2804](info): set up 2 sockets
Jun 20 02:35:38 dist-test-slave-rpg6 krb5kdc[2804](info): commencing operation
krb5kdc: starting...
W20260620 02:35:39.920311 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:35:40 dist-test-slave-rpg6 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:35:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:35:40.133304 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41355
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:40.481001 2817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:40.481400 2817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:40.481551 2817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:40.489382 2817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:35:40.489593 2817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:40.489686 2817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:40.489795 2817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:35:40.489889 2817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:35:40.501369 2817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:40.502931 2817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:40.504411 2817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:40.517760 2822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:40.517728 2823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:40.517786 2825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:40.518661 2817 server_base.cc:1034] running on GCE node
I20260620 02:35:40.519610 2817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:40.521301 2817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:40.522567 2817 hybrid_clock.cc:648] HybridClock initialized: now 1781922940522527 us; error 43 us; skew 500 ppm
I20260620 02:35:40.522970 2817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:40 dist-test-slave-rpg6 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:40.526501 2817 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:35:40.528389 2817 webserver.cc:460] Webserver started at http://127.0.97.62:36361/ using document root <none> and password file <none>
I20260620 02:35:40.529139 2817 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:40.529295 2817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:40.529623 2817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:40.532925 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "8e62153f72dc4792999b69c129f34946"
format_stamp: "Formatted at 2026-06-20 02:35:40 on dist-test-slave-rpg6"
server_key: "f39344c2fe50addd2f2fba54f686dc5d"
server_key_iv: "917ad9619537908e389c321f05f2d033"
server_key_version: "encryptionkey@0"
I20260620 02:35:40.533742 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "8e62153f72dc4792999b69c129f34946"
format_stamp: "Formatted at 2026-06-20 02:35:40 on dist-test-slave-rpg6"
server_key: "f39344c2fe50addd2f2fba54f686dc5d"
server_key_iv: "917ad9619537908e389c321f05f2d033"
server_key_version: "encryptionkey@0"
I20260620 02:35:40.539784 2817 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260620 02:35:40.543789 2832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:40.545429 2817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:35:40.545634 2817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "8e62153f72dc4792999b69c129f34946"
format_stamp: "Formatted at 2026-06-20 02:35:40 on dist-test-slave-rpg6"
server_key: "f39344c2fe50addd2f2fba54f686dc5d"
server_key_iv: "917ad9619537908e389c321f05f2d033"
server_key_version: "encryptionkey@0"
I20260620 02:35:40.545869 2817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:40.568374 2817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:40.573659 2817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:40.573988 2817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:40.595099 2817 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41355
I20260620 02:35:40.595129 2883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41355 every 8 connection(s)
I20260620 02:35:40.596799 2817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:35:40.601454 2884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:40.604358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 2817
I20260620 02:35:40.604660 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:35:40.605497 388 external_mini_cluster.cc:1428] Setting key d9b96ee8d47a87f70505907edcacf677
I20260620 02:35:40.615430 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946: Bootstrap starting.
I20260620 02:35:40.619320 2884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:40.620640 2884 log.cc:824] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:35:40 dist-test-slave-rpg6 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:40.624670 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946: No bootstrap required, opened a new log
I20260620 02:35:40.631210 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:40.608132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.0.97.62:41355)
0620 02:35:40.609174 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:40.609188 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:40.609233 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:35:40.611049 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:40.611077 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:40.611530 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:40.612063 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:40.613674 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:40.615064 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:40.616550 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:40.617309 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:40.617559 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:40.622270 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:40.622283 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:40.622311 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:40.622359 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:40.626379 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:40.627222 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:40.627234 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:40.627246 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:40.627465 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:40.627807 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:40.627813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:40.627816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:40.628583 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:40.628891 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:40.629233 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 02:35:40.629687 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":451,"threads_started":1}
I20260620 02:35:40.635167 2884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e62153f72dc4792999b69c129f34946" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41355 } }
I20260620 02:35:40.635535 2884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:40.635601 2884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e62153f72dc4792999b69c129f34946, State: Initialized, Role: FOLLOWER
I20260620 02:35:40.636215 2884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e62153f72dc4792999b69c129f34946" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41355 } }
I20260620 02:35:40.636461 2884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:35:40.636590 2884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:35:40.636775 2884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:40.638942 2884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e62153f72dc4792999b69c129f34946" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41355 } }
I20260620 02:35:40.639515 2884 leader_election.cc:302] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e62153f72dc4792999b69c129f34946; no voters:
I20260620 02:35:40.640785 2884 leader_election.cc:288] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:35:40.641026 2889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:40.642524 2889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [term 1 LEADER]: Becoming Leader. State: Replica: 8e62153f72dc4792999b69c129f34946, State: Running, Role: LEADER
I20260620 02:35:40.643267 2889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e62153f72dc4792999b69c129f34946" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41355 } }
I20260620 02:35:40.643926 2884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:35:40.646427 2891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e62153f72dc4792999b69c129f34946. Latest consensus state: current_term: 1 leader_uuid: "8e62153f72dc4792999b69c129f34946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e62153f72dc4792999b69c129f34946" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41355 } } }
I20260620 02:35:40.646845 2891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:40.646500 2890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e62153f72dc4792999b69c129f34946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e62153f72dc4792999b69c129f34946" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41355 } } }
I20260620 02:35:40.648777 2890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:40.661391 2896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:35:40.667914 2896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:35:40.679447 2896 catalog_manager.cc:1347] Generated new cluster ID: ecd405064e334b73a3c0caad133674d2
I20260620 02:35:40.679621 2896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:35:40.694626 2896 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:35:40.696412 2896 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:35:40.704224 2896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e62153f72dc4792999b69c129f34946: Generated new TSK 0
I20260620 02:35:40.705523 2896 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
W20260620 02:35:40.839735 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.1: real 0.126s user 0.002s sys 0.012s
I20260620 02:35:40.841015 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41355
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:41.210697 2911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:41.211004 2911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:41.211164 2911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:41.218914 2911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:41.219038 2911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:41.219203 2911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:35:41.229429 2911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:41.230857 2911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:41.232357 2911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:41.246771 2917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:41.246651 2919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:41.247455 2916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:41.248541 2911 server_base.cc:1034] running on GCE node
I20260620 02:35:41.249300 2911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:41.250463 2911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:41.251732 2911 hybrid_clock.cc:648] HybridClock initialized: now 1781922941251666 us; error 55 us; skew 500 ppm
I20260620 02:35:41.252159 2911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:41 dist-test-slave-rpg6 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:41.255875 2911 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:35:41.257642 2911 webserver.cc:460] Webserver started at http://127.0.97.1:43967/ using document root <none> and password file <none>
I20260620 02:35:41.258420 2911 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:41.258581 2911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:41.258917 2911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:41.262235 2911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "3ec6462bcbff4ac5a7269179d32da095"
format_stamp: "Formatted at 2026-06-20 02:35:41 on dist-test-slave-rpg6"
server_key: "d624f5328e44ba986ae862157c32c9fe"
server_key_iv: "bafd4f46ef8508ca266abefa303d2824"
server_key_version: "encryptionkey@0"
I20260620 02:35:41.263079 2911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3ec6462bcbff4ac5a7269179d32da095"
format_stamp: "Formatted at 2026-06-20 02:35:41 on dist-test-slave-rpg6"
server_key: "d624f5328e44ba986ae862157c32c9fe"
server_key_iv: "bafd4f46ef8508ca266abefa303d2824"
server_key_version: "encryptionkey@0"
I20260620 02:35:41.268843 2911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:35:41.272985 2926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:41.275394 2911 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:35:41.275619 2911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "3ec6462bcbff4ac5a7269179d32da095"
format_stamp: "Formatted at 2026-06-20 02:35:41 on dist-test-slave-rpg6"
server_key: "d624f5328e44ba986ae862157c32c9fe"
server_key_iv: "bafd4f46ef8508ca266abefa303d2824"
server_key_version: "encryptionkey@0"
I20260620 02:35:41.275916 2911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:41.300563 2911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:41.305567 2911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:41.305896 2911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:41.306978 2911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:41.308468 2911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:41.308579 2911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:41.308735 2911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:41.308807 2911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:41.344944 2911 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45449
I20260620 02:35:41.344995 3038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45449 every 8 connection(s)
I20260620 02:35:41.346289 2911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:35:41.354925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 2911
I20260620 02:35:41.355159 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:35:41.355935 388 external_mini_cluster.cc:1428] Setting key fc0edf18a46e90b240c2483f5618e3d4
Jun 20 02:35:41 dist-test-slave-rpg6 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:41.370420 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:41.349132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49773 (local address 127.0.97.62:41355)
0620 02:35:41.349641 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:41.349645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:41.350900 (+ 1255us) server_negotiation.cc:407] Connection header received
0620 02:35:41.352890 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:41.352896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:41.352973 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:41.353081 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:41.354925 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:41.356123 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:41.358344 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:41.358972 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:41.359152 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:41.363528 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:41.363537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:41.363543 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:41.363596 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:41.366584 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:41.367724 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:41.367729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:41.367731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:41.367958 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:41.368480 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:41.368484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:41.368486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:41.368749 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:41.368864 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:41.369895 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0620 02:35:41.370086 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":302,"threads_started":1}
I20260620 02:35:41.372104 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:41.349844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41355 (local address 127.0.97.1:49773)
0620 02:35:41.350693 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:41.350739 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:41.352562 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:41.353380 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:41.353389 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:41.353945 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:41.354610 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:41.354627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:41.356350 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:41.356355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:41.358172 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:41.358186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:41.359336 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:41.359342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:41.359568 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:41.360423 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:41.360447 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:41.363227 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:41.366801 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:41.366815 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:41.366839 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:41.367571 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:41.368078 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:41.368086 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:41.368092 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:41.368361 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:41.368907 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:41.368920 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:41.369419 (+ 499us) client_negotiation.cc:769] Sending connection context
0620 02:35:41.369769 (+ 350us) client_negotiation.cc:240] Negotiation successful
0620 02:35:41.370395 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":326,"threads_started":1}
I20260620 02:35:41.373890 3039 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41355
I20260620 02:35:41.374228 3039 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:41.375037 3039 heartbeater.cc:511] Master 127.0.97.62:41355 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:35:41.377719 2849 ts_manager.cc:194] Registered new tserver with Master: 3ec6462bcbff4ac5a7269179d32da095 (127.0.97.1:45449)
I20260620 02:35:41.379419 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:35:41.409439 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41355
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:41.767374 3045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:41.767700 3045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:41.767892 3045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:41.775705 3045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:41.775907 3045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:41.776118 3045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:35:41.786502 3045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:41.788009 3045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:41.789520 3045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:41.808617 3050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:41.808763 3053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:41.808637 3051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:41.809536 3045 server_base.cc:1034] running on GCE node
I20260620 02:35:41.810283 3045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:41.811368 3045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:41.812630 3045 hybrid_clock.cc:648] HybridClock initialized: now 1781922941812570 us; error 58 us; skew 500 ppm
I20260620 02:35:41.813023 3045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:41 dist-test-slave-rpg6 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:41.816538 3045 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:35:41.818226 3045 webserver.cc:460] Webserver started at http://127.0.97.2:41187/ using document root <none> and password file <none>
I20260620 02:35:41.818976 3045 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:41.819139 3045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:41.819473 3045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:41.822616 3045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "db0fc47ab398459e9a4636ed24aa97fa"
format_stamp: "Formatted at 2026-06-20 02:35:41 on dist-test-slave-rpg6"
server_key: "937f9e263486012cef6c759f15782535"
server_key_iv: "886f6ad3a4dbc0f0e0216096144455f5"
server_key_version: "encryptionkey@0"
I20260620 02:35:41.823544 3045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "db0fc47ab398459e9a4636ed24aa97fa"
format_stamp: "Formatted at 2026-06-20 02:35:41 on dist-test-slave-rpg6"
server_key: "937f9e263486012cef6c759f15782535"
server_key_iv: "886f6ad3a4dbc0f0e0216096144455f5"
server_key_version: "encryptionkey@0"
I20260620 02:35:41.829592 3045 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260620 02:35:41.833698 3060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:41.835270 3045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:35:41.835493 3045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "db0fc47ab398459e9a4636ed24aa97fa"
format_stamp: "Formatted at 2026-06-20 02:35:41 on dist-test-slave-rpg6"
server_key: "937f9e263486012cef6c759f15782535"
server_key_iv: "886f6ad3a4dbc0f0e0216096144455f5"
server_key_version: "encryptionkey@0"
I20260620 02:35:41.835740 3045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:41.854527 3045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:41.859436 3045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:41.859786 3045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:41.860926 3045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:41.862375 3045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:41.862497 3045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:41.862655 3045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:41.862742 3045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:41.897967 3045 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39647
I20260620 02:35:41.898032 3172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39647 every 8 connection(s)
I20260620 02:35:41.899682 3045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:35:41.901006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 3045
I20260620 02:35:41.901196 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:35:41.901750 388 external_mini_cluster.cc:1428] Setting key b955b40c1eac2b06c5465fb53f520f1f
Jun 20 02:35:41 dist-test-slave-rpg6 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:35:41.926599 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:41.902876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37991 (local address 127.0.97.62:41355)
0620 02:35:41.903456 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:41.903460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:41.904746 (+ 1286us) server_negotiation.cc:407] Connection header received
0620 02:35:41.908057 (+ 3311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:41.908062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:41.908141 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:41.908275 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:41.910770 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:41.912041 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:41.913850 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:41.914711 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:41.914887 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:41.919165 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:41.919172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:41.919177 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:41.919229 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:41.923117 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:41.924102 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:41.924107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:41.924109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:41.924246 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:41.924767 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:41.924771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:41.924773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:41.925058 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:41.925163 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:41.926065 (+ 902us) server_negotiation.cc:299] Negotiation successful
0620 02:35:41.926262 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":350,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:35:41.927749 3176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:41.903571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41355 (local address 127.0.97.2:37991)
0620 02:35:41.904547 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:41.904608 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:41.907569 (+ 2961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:41.908605 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:41.908622 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:41.909497 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:41.910415 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:41.910451 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:41.912240 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:41.912245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:41.913590 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:41.913605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:41.915077 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:41.915082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:41.915316 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:41.916150 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:41.916179 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:41.918829 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:41.923327 (+ 4498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:41.923336 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:41.923352 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:41.923970 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:41.924379 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:41.924383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:41.924385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:41.924652 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:41.925206 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:41.925214 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:41.925637 (+ 423us) client_negotiation.cc:769] Sending connection context
0620 02:35:41.925936 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 02:35:41.926419 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":411,"threads_started":1}
I20260620 02:35:41.929538 3173 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41355
I20260620 02:35:41.929899 3173 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:41.930745 3173 heartbeater.cc:511] Master 127.0.97.62:41355 requested a full tablet report, sending...
I20260620 02:35:41.932600 2849 ts_manager.cc:194] Registered new tserver with Master: db0fc47ab398459e9a4636ed24aa97fa (127.0.97.2:39647)
I20260620 02:35:41.933439 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:35:41.957137 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41355
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:42.314904 3179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:42.315217 3179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:42.315315 3179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:42.322992 3179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:42.323135 3179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:42.323251 3179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:35:42.334587 3179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:42.336076 3179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:42.337649 3179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:42.354712 3184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:42.355206 3185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:42.355356 3187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:42.356738 3179 server_base.cc:1034] running on GCE node
I20260620 02:35:42.357967 3179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:42.359234 3179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:42.360474 3179 hybrid_clock.cc:648] HybridClock initialized: now 1781922942360421 us; error 41 us; skew 500 ppm
I20260620 02:35:42.360918 3179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:42 dist-test-slave-rpg6 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:42.364773 3179 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:35:42.366734 3179 webserver.cc:460] Webserver started at http://127.0.97.3:37073/ using document root <none> and password file <none>
I20260620 02:35:42.367518 3179 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:42.367669 3179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:42.368065 3179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:42.371304 3179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "51cc5792e26348d2a4ad188c92cbb3b7"
format_stamp: "Formatted at 2026-06-20 02:35:42 on dist-test-slave-rpg6"
server_key: "febc8d33f20696d38f3ec601d01a479d"
server_key_iv: "7eb5a4614ffb7faa180acd89a5776f4d"
server_key_version: "encryptionkey@0"
I20260620 02:35:42.372243 3179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "51cc5792e26348d2a4ad188c92cbb3b7"
format_stamp: "Formatted at 2026-06-20 02:35:42 on dist-test-slave-rpg6"
server_key: "febc8d33f20696d38f3ec601d01a479d"
server_key_iv: "7eb5a4614ffb7faa180acd89a5776f4d"
server_key_version: "encryptionkey@0"
I20260620 02:35:42.377900 3179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:35:42.381950 3194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:42.383625 3179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:35:42.384011 3039 heartbeater.cc:503] Master 127.0.97.62:41355 was elected leader, sending a full tablet report...
I20260620 02:35:42.383944 3179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "51cc5792e26348d2a4ad188c92cbb3b7"
format_stamp: "Formatted at 2026-06-20 02:35:42 on dist-test-slave-rpg6"
server_key: "febc8d33f20696d38f3ec601d01a479d"
server_key_iv: "7eb5a4614ffb7faa180acd89a5776f4d"
server_key_version: "encryptionkey@0"
I20260620 02:35:42.384171 3179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:42.400970 3179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:42.405920 3179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:42.406250 3179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:42.407390 3179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:42.408903 3179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:42.409010 3179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:42.409219 3179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:42.409301 3179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:42.445806 3179 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41215
I20260620 02:35:42.445855 3306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41215 every 8 connection(s)
I20260620 02:35:42.447257 3179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:35:42.449195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 3179
I20260620 02:35:42.449385 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:35:42.450150 388 external_mini_cluster.cc:1428] Setting key d496a719d82cbcf9a514ec2bfa306db7
Jun 20 02:35:42 dist-test-slave-rpg6 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:42.471472 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:42.450776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50687 (local address 127.0.97.62:41355)
0620 02:35:42.452279 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:42.452285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:42.453303 (+ 1018us) server_negotiation.cc:407] Connection header received
0620 02:35:42.455040 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:42.455045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:42.455119 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:42.455247 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:42.457262 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:42.458134 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:42.459986 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:42.460473 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:42.460646 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:42.464941 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:42.464947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:42.464950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:42.464988 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:42.468017 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:42.469061 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:42.469066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:42.469069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:42.469198 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:42.469728 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:42.469732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:42.469735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:42.470001 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:42.470110 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:42.470982 (+ 872us) server_negotiation.cc:299] Negotiation successful
0620 02:35:42.471163 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":1227,"threads_started":1}
I20260620 02:35:42.472611 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:42.451807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41355 (local address 127.0.97.3:50687)
0620 02:35:42.452923 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:42.452973 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:42.454740 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:42.455623 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:42.455634 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:42.456366 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:42.457022 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:42.457042 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:42.458382 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:42.458387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:42.459750 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:42.459764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:42.460710 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:42.460715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:42.460940 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:42.461706 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:42.461739 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:42.464638 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:42.468230 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:42.468249 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:42.468279 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:42.468812 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:42.469402 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:42.469406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:42.469409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:42.469617 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:42.470152 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:42.470160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:42.470532 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 02:35:42.470841 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 02:35:42.471302 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":419,"threads_started":1}
I20260620 02:35:42.474213 3307 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41355
I20260620 02:35:42.474567 3307 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:42.475342 3307 heartbeater.cc:511] Master 127.0.97.62:41355 requested a full tablet report, sending...
I20260620 02:35:42.477037 2849 ts_manager.cc:194] Registered new tserver with Master: 51cc5792e26348d2a4ad188c92cbb3b7 (127.0.97.3:41215)
I20260620 02:35:42.478048 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50687
I20260620 02:35:42.478775 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 02:35:42.513159 3309 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:35:42.510341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.0.97.62:41355)
0620 02:35:42.510627 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:42.510632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:42.510997 (+ 365us) server_negotiation.cc:407] Connection header received
0620 02:35:42.511361 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:42.511366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:42.511438 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:42.511534 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:42.512882 (+ 1348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33738: BlockingRecv error: recv got EOF from 127.0.0.1:33738 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
I20260620 02:35:42.516522 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 2911
W20260620 02:35:42.519886 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 2911: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:45449: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:35:42.520098 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 2911
I20260620 02:35:42.536406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3045
W20260620 02:35:42.539640 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3045: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:39647: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:35:42.539852 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 3045
I20260620 02:35:42.556244 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3179
W20260620 02:35:42.559629 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3179: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:41215: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:35:42.559870 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 3179
I20260620 02:35:42.576191 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 2817
I20260620 02:35:42.577527 2879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:42.682380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 2817
2026-06-20T02:35:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3868 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:35:42 dist-test-slave-rpg6 krb5kdc[3325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:35:42 dist-test-slave-rpg6 krb5kdc[3325](info): set up 2 sockets
Jun 20 02:35:42 dist-test-slave-rpg6 krb5kdc[3325](info): commencing operation
krb5kdc: starting...
W20260620 02:35:43.805989 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:35:43 dist-test-slave-rpg6 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:35:44.033790 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39391
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:44.380136 3338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:44.380458 3338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:44.380548 3338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:44.388195 3338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:35:44.388327 3338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:44.388373 3338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:44.388428 3338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:35:44.388479 3338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:35:44.398741 3338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:44.400139 3338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:44.401505 3338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:44.414666 3344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:44.414669 3346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:44.415238 3343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:44.415477 3338 server_base.cc:1034] running on GCE node
I20260620 02:35:44.416466 3338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:44.418020 3338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:44.419292 3338 hybrid_clock.cc:648] HybridClock initialized: now 1781922944419238 us; error 38 us; skew 500 ppm
I20260620 02:35:44.419730 3338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:44 dist-test-slave-rpg6 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:44.430025 3338 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:35:44.431799 3338 webserver.cc:460] Webserver started at http://127.0.97.62:37619/ using document root <none> and password file <none>
I20260620 02:35:44.432595 3338 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:44.432708 3338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:44.432981 3338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:44.436324 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "a4d67507991f4a5fa4b81c31503bdf61"
format_stamp: "Formatted at 2026-06-20 02:35:44 on dist-test-slave-rpg6"
server_key: "6a7984e74cf3779f2e4670c51643b588"
server_key_iv: "9e2c88e6ce79bf8c1e2b4b01894a122d"
server_key_version: "encryptionkey@0"
I20260620 02:35:44.437157 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "a4d67507991f4a5fa4b81c31503bdf61"
format_stamp: "Formatted at 2026-06-20 02:35:44 on dist-test-slave-rpg6"
server_key: "6a7984e74cf3779f2e4670c51643b588"
server_key_iv: "9e2c88e6ce79bf8c1e2b4b01894a122d"
server_key_version: "encryptionkey@0"
I20260620 02:35:44.442802 3338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:35:44.446885 3353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:44.448664 3338 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:35:44.448855 3338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "a4d67507991f4a5fa4b81c31503bdf61"
format_stamp: "Formatted at 2026-06-20 02:35:44 on dist-test-slave-rpg6"
server_key: "6a7984e74cf3779f2e4670c51643b588"
server_key_iv: "9e2c88e6ce79bf8c1e2b4b01894a122d"
server_key_version: "encryptionkey@0"
I20260620 02:35:44.449044 3338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:44.468209 3338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:44.472824 3338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:44.473078 3338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:44.493896 3338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39391
I20260620 02:35:44.493923 3404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39391 every 8 connection(s)
I20260620 02:35:44.495471 3338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:35:44.499409 3405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:44.505858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 3338
I20260620 02:35:44.506170 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:35:44.506884 388 external_mini_cluster.cc:1428] Setting key 4053aecd66d95db5046c5aef3c699fa2
I20260620 02:35:44.511687 3405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61: Bootstrap starting.
I20260620 02:35:44.516332 3405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:44.517632 3405 log.cc:824] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:44.521016 3405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61: No bootstrap required, opened a new log
Jun 20 02:35:44 dist-test-slave-rpg6 krb5kdc[3325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:44.530427 3405 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d67507991f4a5fa4b81c31503bdf61" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } }
I20260620 02:35:44.530956 3405 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:44.531095 3405 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d67507991f4a5fa4b81c31503bdf61, State: Initialized, Role: FOLLOWER
I20260620 02:35:44.531896 3405 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d67507991f4a5fa4b81c31503bdf61" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } }
I20260620 02:35:44.532231 3405 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:35:44.532395 3405 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:35:44.532591 3405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:44.534471 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:44.509103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.0.97.62:39391)
0620 02:35:44.510090 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:44.510105 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:44.510153 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:35:44.512420 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:44.512450 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:44.513055 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:44.513599 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:44.515602 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:44.517657 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:44.519497 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:44.520297 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:44.520517 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:44.525041 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:44.525053 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:44.525077 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:44.525128 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:44.529261 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:44.530306 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:44.530317 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:44.530327 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:44.530537 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:44.531093 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:44.531099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:44.531102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:44.531814 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:44.532220 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:44.532824 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 02:35:44.533297 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":456,"threads_started":1}
I20260620 02:35:44.535272 3405 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d67507991f4a5fa4b81c31503bdf61" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } }
I20260620 02:35:44.535895 3405 leader_election.cc:302] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4d67507991f4a5fa4b81c31503bdf61; no voters:
I20260620 02:35:44.536571 3405 leader_election.cc:288] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:35:44.536736 3410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:44.538020 3410 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [term 1 LEADER]: Becoming Leader. State: Replica: a4d67507991f4a5fa4b81c31503bdf61, State: Running, Role: LEADER
I20260620 02:35:44.538668 3410 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d67507991f4a5fa4b81c31503bdf61" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } }
I20260620 02:35:44.539090 3405 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:35:44.539685 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:35:44.541771 3411 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4d67507991f4a5fa4b81c31503bdf61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d67507991f4a5fa4b81c31503bdf61" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } } }
I20260620 02:35:44.542163 3411 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:44.545087 3412 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d67507991f4a5fa4b81c31503bdf61. Latest consensus state: current_term: 1 leader_uuid: "a4d67507991f4a5fa4b81c31503bdf61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d67507991f4a5fa4b81c31503bdf61" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } } }
I20260620 02:35:44.545714 3412 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:44.547890 3415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:35:44.559967 3415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:35:44.573869 3415 catalog_manager.cc:1347] Generated new cluster ID: 1e939501184d4505a669acfaa6b16813
I20260620 02:35:44.574014 3415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:35:44.591096 3431 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:35:44.602808 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:44.594684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.0.97.62:39391)
0620 02:35:44.594916 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:44.594923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:44.595645 (+ 722us) server_negotiation.cc:407] Connection header received
0620 02:35:44.595892 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:44.595897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:44.595981 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:44.596100 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:44.597266 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:44.598013 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:44.599325 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:44.599886 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:44.600039 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:44.601260 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:44.601266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:44.601269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:44.601909 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:44.601977 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:44.602008 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:44.602137 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:44.602289 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 02:35:44.602393 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:35:44.607582 3415 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:35:44.609521 3415 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:35:44.618425 3415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4d67507991f4a5fa4b81c31503bdf61: Generated new TSK 0
I20260620 02:35:44.619378 3415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:35:44.630555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3338
I20260620 02:35:44.631781 3400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:44.724273 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 3338
2026-06-20T02:35:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2036 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:35:44 dist-test-slave-rpg6 krb5kdc[3439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:35:44 dist-test-slave-rpg6 krb5kdc[3439](info): set up 2 sockets
Jun 20 02:35:44 dist-test-slave-rpg6 krb5kdc[3439](info): commencing operation
krb5kdc: starting...
W20260620 02:35:45.840485 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:35:45 dist-test-slave-rpg6 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:35:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:35:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:35:46.064731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41805
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:46.413021 3452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:46.413414 3452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:46.413587 3452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:46.421120 3452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:35:46.421309 3452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:46.421427 3452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:46.421540 3452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:35:46.421626 3452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:35:46.431757 3452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:46.433241 3452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:46.434580 3452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:46.447197 3457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:46.447304 3460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:46.447319 3458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:46.448498 3452 server_base.cc:1034] running on GCE node
I20260620 02:35:46.449404 3452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:46.450963 3452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:46.452203 3452 hybrid_clock.cc:648] HybridClock initialized: now 1781922946452157 us; error 46 us; skew 500 ppm
I20260620 02:35:46.452554 3452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:46 dist-test-slave-rpg6 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:46.456070 3452 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:35:46.457685 3452 webserver.cc:460] Webserver started at http://127.0.97.62:44793/ using document root <none> and password file <none>
I20260620 02:35:46.458389 3452 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:46.458537 3452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:46.458855 3452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:46.462028 3452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "25c01f2e03c54467bba1ead215fc41a8"
format_stamp: "Formatted at 2026-06-20 02:35:46 on dist-test-slave-rpg6"
server_key: "7e85c4fc88c93dbf2b62bd2356cd0e7b"
server_key_iv: "940f3b7d7ba54ae9d47b4769adb21590"
server_key_version: "encryptionkey@0"
I20260620 02:35:46.462831 3452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "25c01f2e03c54467bba1ead215fc41a8"
format_stamp: "Formatted at 2026-06-20 02:35:46 on dist-test-slave-rpg6"
server_key: "7e85c4fc88c93dbf2b62bd2356cd0e7b"
server_key_iv: "940f3b7d7ba54ae9d47b4769adb21590"
server_key_version: "encryptionkey@0"
I20260620 02:35:46.468360 3452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:35:46.472090 3467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:46.473506 3452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:35:46.473742 3452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "25c01f2e03c54467bba1ead215fc41a8"
format_stamp: "Formatted at 2026-06-20 02:35:46 on dist-test-slave-rpg6"
server_key: "7e85c4fc88c93dbf2b62bd2356cd0e7b"
server_key_iv: "940f3b7d7ba54ae9d47b4769adb21590"
server_key_version: "encryptionkey@0"
I20260620 02:35:46.473958 3452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:46.494891 3452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:46.499714 3452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:46.500089 3452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:46.519970 3452 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41805
I20260620 02:35:46.520033 3518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41805 every 8 connection(s)
I20260620 02:35:46.521471 3452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:35:46.525426 3519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:46.527565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 3452
I20260620 02:35:46.527869 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:35:46.528661 388 external_mini_cluster.cc:1428] Setting key 54afeed6a2e31795014897097ce72451
I20260620 02:35:46.539258 3519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8: Bootstrap starting.
I20260620 02:35:46.543605 3519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8: Neither blocks nor log segments found. Creating new log.
Jun 20 02:35:46 dist-test-slave-rpg6 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:46.545084 3519 log.cc:824] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:46.548825 3519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8: No bootstrap required, opened a new log
I20260620 02:35:46.554689 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:46.530949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.97.62:41805)
0620 02:35:46.531878 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:46.531910 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:46.531964 (+ 54us) server_negotiation.cc:407] Connection header received
0620 02:35:46.533968 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:46.533997 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:46.534457 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:46.534833 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:46.536776 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:46.538273 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:46.540181 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:46.540912 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:46.541100 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:46.546385 (+ 5285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:46.546410 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:46.546435 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:46.546499 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:46.549898 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:46.550935 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:46.550943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:46.550950 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:46.551157 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:46.551605 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:46.551609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:46.551611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:46.552338 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:46.552581 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:46.553048 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 02:35:46.553531 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":382,"threads_started":1}
I20260620 02:35:46.558787 3519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c01f2e03c54467bba1ead215fc41a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } }
I20260620 02:35:46.559252 3519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:46.559372 3519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25c01f2e03c54467bba1ead215fc41a8, State: Initialized, Role: FOLLOWER
I20260620 02:35:46.560050 3519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c01f2e03c54467bba1ead215fc41a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } }
I20260620 02:35:46.560310 3519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:35:46.560460 3519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:35:46.560636 3519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:46.563783 3519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c01f2e03c54467bba1ead215fc41a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } }
I20260620 02:35:46.564561 3519 leader_election.cc:302] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25c01f2e03c54467bba1ead215fc41a8; no voters:
I20260620 02:35:46.565387 3519 leader_election.cc:288] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:35:46.565532 3524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:46.568868 3519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:35:46.572643 3524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [term 1 LEADER]: Becoming Leader. State: Replica: 25c01f2e03c54467bba1ead215fc41a8, State: Running, Role: LEADER
I20260620 02:35:46.573493 3524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c01f2e03c54467bba1ead215fc41a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } }
I20260620 02:35:46.577632 3525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25c01f2e03c54467bba1ead215fc41a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c01f2e03c54467bba1ead215fc41a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } } }
I20260620 02:35:46.578377 3525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:46.579514 3532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:35:46.579448 3526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25c01f2e03c54467bba1ead215fc41a8. Latest consensus state: current_term: 1 leader_uuid: "25c01f2e03c54467bba1ead215fc41a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c01f2e03c54467bba1ead215fc41a8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } } }
I20260620 02:35:46.579906 3526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:46.586954 3532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:35:46.610905 3532 catalog_manager.cc:1347] Generated new cluster ID: 90d9a005798a491684f24a2429e1b412
I20260620 02:35:46.611048 3532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:35:46.622817 3532 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:35:46.624756 3532 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:35:46.634356 3532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25c01f2e03c54467bba1ead215fc41a8: Generated new TSK 0
I20260620 02:35:46.635525 3532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:35:46.696168 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41805
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:47.053086 3545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:47.053416 3545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:47.053579 3545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:47.062789 3545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:47.062933 3545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:47.063055 3545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:35:47.075507 3545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:47.076967 3545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:47.078521 3545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:47.093171 3550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:47.093643 3551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:47.093714 3553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:47.095206 3545 server_base.cc:1034] running on GCE node
I20260620 02:35:47.096024 3545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:47.097153 3545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:47.098434 3545 hybrid_clock.cc:648] HybridClock initialized: now 1781922947098387 us; error 37 us; skew 500 ppm
I20260620 02:35:47.098848 3545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:47 dist-test-slave-rpg6 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:47.103091 3545 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:35:47.105254 3545 webserver.cc:460] Webserver started at http://127.0.97.1:44517/ using document root <none> and password file <none>
I20260620 02:35:47.106083 3545 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:47.106238 3545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:47.106560 3545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:47.109826 3545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "a468fe58f81e4cc5b1f7526e18bd96c9"
format_stamp: "Formatted at 2026-06-20 02:35:47 on dist-test-slave-rpg6"
server_key: "de6ff5231a048e3b2e7a3dcd09773611"
server_key_iv: "dc73993c56a5aa36fa8ead7dbe1f112d"
server_key_version: "encryptionkey@0"
I20260620 02:35:47.110680 3545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a468fe58f81e4cc5b1f7526e18bd96c9"
format_stamp: "Formatted at 2026-06-20 02:35:47 on dist-test-slave-rpg6"
server_key: "de6ff5231a048e3b2e7a3dcd09773611"
server_key_iv: "dc73993c56a5aa36fa8ead7dbe1f112d"
server_key_version: "encryptionkey@0"
I20260620 02:35:47.116626 3545 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 02:35:47.120612 3560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:47.122247 3545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:35:47.122455 3545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "a468fe58f81e4cc5b1f7526e18bd96c9"
format_stamp: "Formatted at 2026-06-20 02:35:47 on dist-test-slave-rpg6"
server_key: "de6ff5231a048e3b2e7a3dcd09773611"
server_key_iv: "dc73993c56a5aa36fa8ead7dbe1f112d"
server_key_version: "encryptionkey@0"
I20260620 02:35:47.122690 3545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:47.140249 3545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:47.144863 3545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:47.145200 3545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:47.146216 3545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:47.147655 3545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:47.147764 3545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:47.147953 3545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:47.148020 3545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:47.183874 3545 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45131
I20260620 02:35:47.183902 3672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45131 every 8 connection(s)
I20260620 02:35:47.185245 3545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:35:47.187988 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 3545
I20260620 02:35:47.188217 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:35:47.189031 388 external_mini_cluster.cc:1428] Setting key f445df09302ea411045017e7235d1c3b
Jun 20 02:35:47 dist-test-slave-rpg6 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:35:47.208217 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:47.188219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55265 (local address 127.0.97.62:41805)
0620 02:35:47.188719 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:47.188723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:47.190219 (+ 1496us) server_negotiation.cc:407] Connection header received
0620 02:35:47.192198 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:47.192203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:47.192295 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:47.192426 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:47.194046 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:47.194785 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:47.196554 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:47.196958 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:47.197097 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:47.201118 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:47.201124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:47.201127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:47.201164 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:47.204357 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:47.205266 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:47.205270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:47.205273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:47.205399 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:47.205902 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:47.205912 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:47.205915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:47.206201 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:47.206363 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:47.207552 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0620 02:35:47.207811 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":256,"threads_started":1}
I20260620 02:35:47.209517 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:47.189017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41805 (local address 127.0.97.1:55265)
0620 02:35:47.189871 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:47.189966 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:47.191902 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:47.192690 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:47.192700 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:47.193252 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:47.193853 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:47.193872 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:47.194974 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:47.194978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:47.196380 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:47.196394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:47.197214 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:47.197221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:47.197460 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:47.198240 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:47.198264 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:47.200834 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:47.204559 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:47.204570 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:47.204597 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:47.205106 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:47.205539 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:47.205544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:47.205548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:47.205775 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:47.206339 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:47.206348 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:47.207049 (+ 701us) client_negotiation.cc:769] Sending connection context
0620 02:35:47.207434 (+ 385us) client_negotiation.cc:240] Negotiation successful
0620 02:35:47.208115 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":362,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:35:47.211665 3673 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41805
I20260620 02:35:47.212065 3673 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:47.212949 3673 heartbeater.cc:511] Master 127.0.97.62:41805 requested a full tablet report, sending...
I20260620 02:35:47.215176 3484 ts_manager.cc:194] Registered new tserver with Master: a468fe58f81e4cc5b1f7526e18bd96c9 (127.0.97.1:45131)
I20260620 02:35:47.217048 3484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:35:47.237195 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41805
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:47.583289 3679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:47.583603 3679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:47.583772 3679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:47.591641 3679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:47.591789 3679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:47.592053 3679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:35:47.602450 3679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:47.603967 3679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:47.605724 3679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:47.620360 3684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:47.620270 3685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:47.620397 3687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:47.621064 3679 server_base.cc:1034] running on GCE node
I20260620 02:35:47.621775 3679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:47.622813 3679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:47.624046 3679 hybrid_clock.cc:648] HybridClock initialized: now 1781922947623987 us; error 57 us; skew 500 ppm
I20260620 02:35:47.624393 3679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:47 dist-test-slave-rpg6 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:47.628301 3679 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:35:47.630086 3679 webserver.cc:460] Webserver started at http://127.0.97.2:46119/ using document root <none> and password file <none>
I20260620 02:35:47.630853 3679 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:47.630959 3679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:47.631318 3679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:47.634361 3679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "fa02851803454becada76fade3a0b7a9"
format_stamp: "Formatted at 2026-06-20 02:35:47 on dist-test-slave-rpg6"
server_key: "a56ef3692a91135aaed51884f8d62684"
server_key_iv: "52b3f74b4a3bd90295847ff5ea36bac7"
server_key_version: "encryptionkey@0"
I20260620 02:35:47.635181 3679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fa02851803454becada76fade3a0b7a9"
format_stamp: "Formatted at 2026-06-20 02:35:47 on dist-test-slave-rpg6"
server_key: "a56ef3692a91135aaed51884f8d62684"
server_key_iv: "52b3f74b4a3bd90295847ff5ea36bac7"
server_key_version: "encryptionkey@0"
I20260620 02:35:47.640704 3679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 02:35:47.644486 3694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:47.646126 3679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:35:47.646304 3679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "fa02851803454becada76fade3a0b7a9"
format_stamp: "Formatted at 2026-06-20 02:35:47 on dist-test-slave-rpg6"
server_key: "a56ef3692a91135aaed51884f8d62684"
server_key_iv: "52b3f74b4a3bd90295847ff5ea36bac7"
server_key_version: "encryptionkey@0"
I20260620 02:35:47.646513 3679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:47.660893 3679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:47.665759 3679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:47.666090 3679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:47.667145 3679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:47.668727 3679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:47.668808 3679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:47.668951 3679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:47.669029 3679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:47.705442 3679 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43723
I20260620 02:35:47.705497 3806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43723 every 8 connection(s)
I20260620 02:35:47.706911 3679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:35:47.708936 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 3679
I20260620 02:35:47.709160 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:35:47.709738 388 external_mini_cluster.cc:1428] Setting key 8f44d94300bb397084ff32aed2fc0cae
Jun 20 02:35:47 dist-test-slave-rpg6 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:35:47.732069 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:47.709843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51583 (local address 127.0.97.62:41805)
0620 02:35:47.710610 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:47.710614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:47.711552 (+ 938us) server_negotiation.cc:407] Connection header received
0620 02:35:47.713605 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:47.713611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:47.713699 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:47.713854 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:47.715885 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:47.716932 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:47.719672 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:47.720285 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:47.720396 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:47.725150 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:47.725156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:47.725160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:47.725196 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:47.728351 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:47.729419 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:47.729423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:47.729425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:47.729551 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:47.730191 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:47.730196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:47.730198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:47.730576 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:47.730694 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:47.731520 (+ 826us) server_negotiation.cc:299] Negotiation successful
0620 02:35:47.731727 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":533,"threads_started":1}
I20260620 02:35:47.733258 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:47.710410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41805 (local address 127.0.97.2:51583)
0620 02:35:47.711364 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:47.711414 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:47.713315 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:47.714115 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:47.714124 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:47.714666 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:47.715640 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:47.715663 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:47.717113 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:47.717121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:47.719475 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:47.719495 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:47.720528 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:47.720537 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:47.720870 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:47.721649 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:47.721683 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:47.724839 (+ 3156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:47.728542 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:47.728554 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:47.728580 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:47.729292 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:47.729690 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:47.729695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:47.729700 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:47.730002 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:47.730743 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:47.730750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:47.731107 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 02:35:47.731414 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 02:35:47.731952 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":374,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:35:47.734972 3807 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41805
I20260620 02:35:47.735366 3807 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:47.736403 3807 heartbeater.cc:511] Master 127.0.97.62:41805 requested a full tablet report, sending...
I20260620 02:35:47.738369 3484 ts_manager.cc:194] Registered new tserver with Master: fa02851803454becada76fade3a0b7a9 (127.0.97.2:43723)
I20260620 02:35:47.739466 3484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:35:47.760525 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41805
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:48.122171 3813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:48.122491 3813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:48.122601 3813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:48.130460 3813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:48.130626 3813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:48.130951 3813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:35:48.141688 3813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:48.143002 3813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:48.144414 3813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:48.158522 3819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:48.158617 3818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:48.158574 3821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:48.160230 3813 server_base.cc:1034] running on GCE node
I20260620 02:35:48.161257 3813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:48.162842 3813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:48.166280 3813 hybrid_clock.cc:648] HybridClock initialized: now 1781922948166185 us; error 72 us; skew 500 ppm
I20260620 02:35:48.166795 3813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:48 dist-test-slave-rpg6 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:48.170610 3813 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:35:48.172592 3813 webserver.cc:460] Webserver started at http://127.0.97.3:34577/ using document root <none> and password file <none>
I20260620 02:35:48.173388 3813 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:48.173509 3813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:48.173887 3813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:48.178177 3813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "b368cf760baa44d9b82179da6cb49f0a"
format_stamp: "Formatted at 2026-06-20 02:35:48 on dist-test-slave-rpg6"
server_key: "b9545f64901e085d2daf5c37ac81081c"
server_key_iv: "fb327487e4b677c2962f70fcea0f4341"
server_key_version: "encryptionkey@0"
I20260620 02:35:48.179117 3813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b368cf760baa44d9b82179da6cb49f0a"
format_stamp: "Formatted at 2026-06-20 02:35:48 on dist-test-slave-rpg6"
server_key: "b9545f64901e085d2daf5c37ac81081c"
server_key_iv: "fb327487e4b677c2962f70fcea0f4341"
server_key_version: "encryptionkey@0"
I20260620 02:35:48.185757 3813 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 02:35:48.190385 3828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:48.192364 3813 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:35:48.192591 3813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "b368cf760baa44d9b82179da6cb49f0a"
format_stamp: "Formatted at 2026-06-20 02:35:48 on dist-test-slave-rpg6"
server_key: "b9545f64901e085d2daf5c37ac81081c"
server_key_iv: "fb327487e4b677c2962f70fcea0f4341"
server_key_version: "encryptionkey@0"
I20260620 02:35:48.192821 3813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:48.212433 3813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:48.217326 3813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:48.217640 3813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:48.218680 3813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:48.220139 3813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:48.220254 3813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:48.220429 3813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:48.220506 3813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:48.220700 3673 heartbeater.cc:503] Master 127.0.97.62:41805 was elected leader, sending a full tablet report...
I20260620 02:35:48.262034 3813 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42375
I20260620 02:35:48.262089 3940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42375 every 8 connection(s)
I20260620 02:35:48.263491 3813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:35:48.269227 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 3813
I20260620 02:35:48.269485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:35:48.270076 388 external_mini_cluster.cc:1428] Setting key 937e754eba3422770785761d86ab2236
Jun 20 02:35:48 dist-test-slave-rpg6 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:48.286682 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:48.266434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60683 (local address 127.0.97.62:41805)
0620 02:35:48.266983 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:48.266988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:48.267970 (+ 982us) server_negotiation.cc:407] Connection header received
0620 02:35:48.270087 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:48.270096 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:48.270210 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:48.270346 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:48.272297 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.273193 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.274932 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.275380 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.275512 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:48.279469 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:48.279475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:48.279478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:48.279515 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:48.282736 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:48.283665 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:48.283669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:48.283672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:48.283897 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:48.284539 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:48.284544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:48.284547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:48.284897 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:48.285070 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:48.286042 (+ 972us) server_negotiation.cc:299] Negotiation successful
0620 02:35:48.286303 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":315,"threads_started":1}
I20260620 02:35:48.287900 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:48.266995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41805 (local address 127.0.97.3:60683)
0620 02:35:48.267745 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:48.267793 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:48.269703 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:48.270572 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:48.270582 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:48.271262 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:48.272084 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:48.272113 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.273406 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.273411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:48.274757 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:48.274772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.275647 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.275652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:48.275976 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:48.276729 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:48.276757 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:48.279216 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:48.282937 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:48.282946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:48.282964 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:48.283478 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:48.284043 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:48.284049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:48.284052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:48.284332 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:48.285044 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:48.285053 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:48.285564 (+ 511us) client_negotiation.cc:769] Sending connection context
0620 02:35:48.285894 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 02:35:48.286463 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":289,"threads_started":1}
I20260620 02:35:48.289853 3941 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41805
I20260620 02:35:48.290298 3941 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:48.291461 3941 heartbeater.cc:511] Master 127.0.97.62:41805 requested a full tablet report, sending...
I20260620 02:35:48.293588 3484 ts_manager.cc:194] Registered new tserver with Master: b368cf760baa44d9b82179da6cb49f0a (127.0.97.3:42375)
I20260620 02:35:48.294965 3484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60683
I20260620 02:35:48.300621 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:35:48 dist-test-slave-rpg6 krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922948, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 02:35:48 dist-test-slave-rpg6 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922948, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:48.353641 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:48.338127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.0.97.62:41805)
0620 02:35:48.338395 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:48.338400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:48.338585 (+ 185us) server_negotiation.cc:407] Connection header received
0620 02:35:48.338774 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:48.338779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:48.338892 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:48.339022 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:48.340617 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.341440 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.342798 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.343218 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.343334 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:48.347557 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:48.347563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:48.347567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:48.347611 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:48.350464 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:48.351276 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:48.351280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:48.351282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:48.351451 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:48.352017 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:48.352021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:48.352023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:48.352434 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:48.352693 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:48.353040 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 02:35:48.353308 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
W20260620 02:35:48.354286 3484 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35428
I20260620 02:35:48.356582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3545
Jun 20 02:35:48 dist-test-slave-rpg6 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922948, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:35:48.374786 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:48.357864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.0.97.1:45131)
0620 02:35:48.358422 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:48.358429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:48.358454 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:35:48.358511 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:48.358518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:48.358760 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:48.358887 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:48.360094 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.361109 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.362547 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.363170 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.363275 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:48.367936 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:48.367959 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:48.367980 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:48.368024 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:48.371318 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:48.372262 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:48.372271 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:48.372273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:48.372414 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:48.373232 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:48.373236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:48.373238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:48.373626 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:48.373861 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:48.374258 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 02:35:48.374489 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":273,"threads_started":1}
W20260620 02:35:48.376181 3668 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48064
W20260620 02:35:48.377498 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3545: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:35:48.377687 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 3545
I20260620 02:35:48.394076 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3679
Jun 20 02:35:48 dist-test-slave-rpg6 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922948, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:35:48.411056 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:48.394973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.0.97.2:43723)
0620 02:35:48.395576 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:48.395584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:48.395617 (+ 33us) server_negotiation.cc:407] Connection header received
0620 02:35:48.395683 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:48.395689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:48.395983 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:48.396115 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:48.397494 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.398602 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.399782 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.400401 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.400504 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:48.404749 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:48.404763 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:48.404780 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:48.404819 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:48.408364 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:48.409133 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:48.409142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:48.409144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:48.409284 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:48.409681 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:48.409684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:48.409686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:48.410043 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:48.410356 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:48.410520 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 02:35:48.410759 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":355,"threads_started":1}
W20260620 02:35:48.412429 3802 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33582
W20260620 02:35:48.413560 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3679: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:35:48.413727 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 3679
I20260620 02:35:48.429749 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3813
Jun 20 02:35:48 dist-test-slave-rpg6 krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922948, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:35:48.447575 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:48.430914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.0.97.3:42375)
0620 02:35:48.431656 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:48.431663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:48.431692 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:35:48.431754 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:48.431762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:48.432050 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:48.432294 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:48.433617 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.434715 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.436101 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:48.436728 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:48.436887 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:48.441503 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:48.441519 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:48.441535 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:48.441580 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:48.444558 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:48.445348 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:48.445356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:48.445359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:48.445488 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:48.445878 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:48.445884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:48.445887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:48.446273 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:48.446586 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:48.446913 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 02:35:48.447209 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":418,"threads_started":1}
W20260620 02:35:48.449169 3936 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35108
W20260620 02:35:48.450497 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3813: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:35:48.450677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 3813
I20260620 02:35:48.467439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3452
I20260620 02:35:48.468765 3514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:48.574196 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 3452
2026-06-20T02:35:48Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3851 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:35:48 dist-test-slave-rpg6 krb5kdc[3959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:35:48 dist-test-slave-rpg6 krb5kdc[3959](info): set up 2 sockets
Jun 20 02:35:48 dist-test-slave-rpg6 krb5kdc[3959](info): commencing operation
krb5kdc: starting...
W20260620 02:35:49.688475 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:35:49 dist-test-slave-rpg6 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:35:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:35:49.925773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37481
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:50.281316 3972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:50.281639 3972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:50.281731 3972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:50.291402 3972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:35:50.291576 3972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:50.291674 3972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:50.291767 3972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:35:50.291857 3972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:35:50.303560 3972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:50.305058 3972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:50.306500 3972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:50.319921 3977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:50.319931 3978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:50.320016 3980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:50.320981 3972 server_base.cc:1034] running on GCE node
I20260620 02:35:50.321928 3972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:50.323518 3972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:50.324790 3972 hybrid_clock.cc:648] HybridClock initialized: now 1781922950324709 us; error 67 us; skew 500 ppm
I20260620 02:35:50.325212 3972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:50 dist-test-slave-rpg6 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:50.333500 3972 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:35:50.335425 3972 webserver.cc:460] Webserver started at http://127.0.97.62:35109/ using document root <none> and password file <none>
I20260620 02:35:50.336313 3972 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:50.336459 3972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:50.336825 3972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:50.340701 3972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "c52eef3979e7427fb6e2f15afb59ef3f"
format_stamp: "Formatted at 2026-06-20 02:35:50 on dist-test-slave-rpg6"
server_key: "d556d0502b65cba5301b425d66b85bcd"
server_key_iv: "a8216bd7c625020e16feb87f88fe8f67"
server_key_version: "encryptionkey@0"
I20260620 02:35:50.341768 3972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "c52eef3979e7427fb6e2f15afb59ef3f"
format_stamp: "Formatted at 2026-06-20 02:35:50 on dist-test-slave-rpg6"
server_key: "d556d0502b65cba5301b425d66b85bcd"
server_key_iv: "a8216bd7c625020e16feb87f88fe8f67"
server_key_version: "encryptionkey@0"
I20260620 02:35:50.348871 3972 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 02:35:50.353121 3987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:50.354704 3972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:35:50.354933 3972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "c52eef3979e7427fb6e2f15afb59ef3f"
format_stamp: "Formatted at 2026-06-20 02:35:50 on dist-test-slave-rpg6"
server_key: "d556d0502b65cba5301b425d66b85bcd"
server_key_iv: "a8216bd7c625020e16feb87f88fe8f67"
server_key_version: "encryptionkey@0"
I20260620 02:35:50.355156 3972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:50.379874 3972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:50.385438 3972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:50.385802 3972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:50.407299 3972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37481
I20260620 02:35:50.407362 4038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37481 every 8 connection(s)
I20260620 02:35:50.408908 3972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:35:50.413089 4039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:50.419255 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 3972
I20260620 02:35:50.419737 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:35:50.420460 388 external_mini_cluster.cc:1428] Setting key ff7cfa7a014fe18f1a3168774c9271e7
I20260620 02:35:50.425289 4039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f: Bootstrap starting.
I20260620 02:35:50.429639 4039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:50.431265 4039 log.cc:824] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:50.435034 4039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f: No bootstrap required, opened a new log
Jun 20 02:35:50 dist-test-slave-rpg6 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:50.446278 4039 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52eef3979e7427fb6e2f15afb59ef3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37481 } }
I20260620 02:35:50.446888 4039 raft_consensus.cc:374] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:50.447014 4039 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52eef3979e7427fb6e2f15afb59ef3f, State: Initialized, Role: FOLLOWER
I20260620 02:35:50.447814 4039 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52eef3979e7427fb6e2f15afb59ef3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37481 } }
I20260620 02:35:50.448145 4039 raft_consensus.cc:388] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:35:50.448287 4039 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:35:50.448488 4039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:50.448550 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:50.422770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59088 (local address 127.0.97.62:37481)
0620 02:35:50.423651 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:50.423689 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:50.423744 (+ 55us) server_negotiation.cc:407] Connection header received
0620 02:35:50.425692 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:50.425737 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:50.426243 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:50.426629 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:50.428668 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:50.430245 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:50.432224 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:50.433285 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:50.433496 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:50.438161 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:50.438174 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:50.438201 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:50.438271 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:50.443035 (+ 4764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:50.444058 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:50.444070 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:50.444084 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:50.444376 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:50.444945 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:50.444951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:50.444956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:50.445773 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:50.446069 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:50.446505 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 02:35:50.447019 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":410,"threads_started":1}
I20260620 02:35:50.451023 4039 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52eef3979e7427fb6e2f15afb59ef3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37481 } }
I20260620 02:35:50.451601 4039 leader_election.cc:302] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c52eef3979e7427fb6e2f15afb59ef3f; no voters:
I20260620 02:35:50.452276 4039 leader_election.cc:288] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:35:50.452437 4044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:50.453189 4044 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [term 1 LEADER]: Becoming Leader. State: Replica: c52eef3979e7427fb6e2f15afb59ef3f, State: Running, Role: LEADER
I20260620 02:35:50.453735 4044 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52eef3979e7427fb6e2f15afb59ef3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37481 } }
I20260620 02:35:50.454471 4039 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:35:50.457182 4046 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52eef3979e7427fb6e2f15afb59ef3f. Latest consensus state: current_term: 1 leader_uuid: "c52eef3979e7427fb6e2f15afb59ef3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52eef3979e7427fb6e2f15afb59ef3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37481 } } }
I20260620 02:35:50.457659 4046 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:50.460103 4045 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c52eef3979e7427fb6e2f15afb59ef3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52eef3979e7427fb6e2f15afb59ef3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37481 } } }
I20260620 02:35:50.460636 4045 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:50.462582 4053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:35:50.468343 4053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:35:50.478633 4053 catalog_manager.cc:1347] Generated new cluster ID: b2c622ceef0a4d9584aafb430254ada2
I20260620 02:35:50.478794 4053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:35:50.512113 4053 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:35:50.514174 4053 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:35:50.523525 4053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c52eef3979e7427fb6e2f15afb59ef3f: Generated new TSK 0
I20260620 02:35:50.524549 4053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:35:50.594132 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37481
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:50.995442 4065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:50.995752 4065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:50.996001 4065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:51.004467 4065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:51.004611 4065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:51.004735 4065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:35:51.015758 4065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:51.017962 4065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:51.019703 4065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:51.034435 4070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:51.034358 4071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:51.034456 4073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:51.035847 4065 server_base.cc:1034] running on GCE node
I20260620 02:35:51.036592 4065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:51.037679 4065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:51.038960 4065 hybrid_clock.cc:648] HybridClock initialized: now 1781922951038920 us; error 39 us; skew 500 ppm
I20260620 02:35:51.039379 4065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:51 dist-test-slave-rpg6 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:51.043161 4065 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:35:51.044929 4065 webserver.cc:460] Webserver started at http://127.0.97.1:39365/ using document root <none> and password file <none>
I20260620 02:35:51.045718 4065 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:51.045862 4065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:51.046233 4065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:51.049614 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "8876faa349be444e90361c4ec03f73d5"
format_stamp: "Formatted at 2026-06-20 02:35:51 on dist-test-slave-rpg6"
server_key: "d08e65a0e189ee5486e2071e1d1cacc3"
server_key_iv: "ef05075bb7b98aa0a8f572a2ed1d309f"
server_key_version: "encryptionkey@0"
I20260620 02:35:51.050524 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8876faa349be444e90361c4ec03f73d5"
format_stamp: "Formatted at 2026-06-20 02:35:51 on dist-test-slave-rpg6"
server_key: "d08e65a0e189ee5486e2071e1d1cacc3"
server_key_iv: "ef05075bb7b98aa0a8f572a2ed1d309f"
server_key_version: "encryptionkey@0"
I20260620 02:35:51.056756 4065 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:35:51.062343 4080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:51.064147 4065 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:35:51.064394 4065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "8876faa349be444e90361c4ec03f73d5"
format_stamp: "Formatted at 2026-06-20 02:35:51 on dist-test-slave-rpg6"
server_key: "d08e65a0e189ee5486e2071e1d1cacc3"
server_key_iv: "ef05075bb7b98aa0a8f572a2ed1d309f"
server_key_version: "encryptionkey@0"
I20260620 02:35:51.064620 4065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:51.080298 4065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:51.087965 4065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:51.088454 4065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:51.090487 4065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:51.092873 4065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:51.092995 4065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:51.093194 4065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:51.093277 4065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:51.140266 4065 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41489
I20260620 02:35:51.140296 4192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41489 every 8 connection(s)
I20260620 02:35:51.141702 4065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:35:51.152122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 4065
I20260620 02:35:51.152331 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:35:51.152964 388 external_mini_cluster.cc:1428] Setting key faa44f8acba3c47eacc82d34373686e9
Jun 20 02:35:51 dist-test-slave-rpg6 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:51.165565 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:51.144707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39965 (local address 127.0.97.62:37481)
0620 02:35:51.145189 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:51.145194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:51.146293 (+ 1099us) server_negotiation.cc:407] Connection header received
0620 02:35:51.148331 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:51.148336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:51.148442 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:51.148562 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:51.150312 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:51.151078 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:51.152937 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:51.153436 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:51.153578 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:51.158354 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:51.158361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:51.158364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:51.158401 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:51.162172 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:51.163072 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:51.163078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:51.163082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:51.163215 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:51.163679 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:51.163684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:51.163687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:51.164052 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:51.164215 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:51.164964 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 02:35:51.165200 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":252,"threads_started":1}
I20260620 02:35:51.166666 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:51.145257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37481 (local address 127.0.97.1:39965)
0620 02:35:51.146026 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:51.146072 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:51.147999 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:51.148835 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:51.148844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:51.149410 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:51.150089 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:51.150109 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:51.151251 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:51.151256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:51.152775 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:51.152788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:51.153679 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:51.153687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:51.153938 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:51.154871 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:51.154896 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:51.158027 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:51.162396 (+ 4369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:51.162414 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:51.162430 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:51.162921 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:51.163340 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:51.163344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:51.163348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:51.163553 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:51.164187 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:51.164196 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:51.164568 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 02:35:51.164841 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 02:35:51.165430 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":313,"threads_started":1}
I20260620 02:35:51.168576 4193 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37481
I20260620 02:35:51.168968 4193 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:51.170037 4193 heartbeater.cc:511] Master 127.0.97.62:37481 requested a full tablet report, sending...
I20260620 02:35:51.173197 4004 ts_manager.cc:194] Registered new tserver with Master: 8876faa349be444e90361c4ec03f73d5 (127.0.97.1:41489)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:35:51.174865 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:35:51.209342 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37481
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:51.574571 4199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:51.574894 4199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:51.574990 4199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:51.582499 4199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:51.582670 4199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:51.582791 4199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:35:51.592788 4199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:51.594113 4199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:51.595425 4199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:51.610215 4204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:51.610296 4205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:51.610323 4207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:51.611557 4199 server_base.cc:1034] running on GCE node
I20260620 02:35:51.612437 4199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:51.613471 4199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:51.614704 4199 hybrid_clock.cc:648] HybridClock initialized: now 1781922951614609 us; error 90 us; skew 500 ppm
I20260620 02:35:51.615031 4199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:51 dist-test-slave-rpg6 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:51.618868 4199 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:35:51.620904 4199 webserver.cc:460] Webserver started at http://127.0.97.2:34367/ using document root <none> and password file <none>
I20260620 02:35:51.621708 4199 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:51.621868 4199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:51.622223 4199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:51.625532 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "d998e601411b44b884686501586bf269"
format_stamp: "Formatted at 2026-06-20 02:35:51 on dist-test-slave-rpg6"
server_key: "22a54c02e69977d756dea95e642ea0ca"
server_key_iv: "fe7cb0127bf8c30219325f03c061d4a7"
server_key_version: "encryptionkey@0"
I20260620 02:35:51.626487 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d998e601411b44b884686501586bf269"
format_stamp: "Formatted at 2026-06-20 02:35:51 on dist-test-slave-rpg6"
server_key: "22a54c02e69977d756dea95e642ea0ca"
server_key_iv: "fe7cb0127bf8c30219325f03c061d4a7"
server_key_version: "encryptionkey@0"
I20260620 02:35:51.632683 4199 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:35:51.636811 4214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:51.638600 4199 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:35:51.638850 4199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "d998e601411b44b884686501586bf269"
format_stamp: "Formatted at 2026-06-20 02:35:51 on dist-test-slave-rpg6"
server_key: "22a54c02e69977d756dea95e642ea0ca"
server_key_iv: "fe7cb0127bf8c30219325f03c061d4a7"
server_key_version: "encryptionkey@0"
I20260620 02:35:51.639113 4199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:51.663780 4199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:51.669188 4199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:51.669523 4199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:51.670608 4199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:51.672291 4199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:51.672637 4199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:51.672857 4199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:51.672945 4199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:51.711987 4199 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45311
I20260620 02:35:51.712087 4326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45311 every 8 connection(s)
I20260620 02:35:51.713492 4199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:35:51.713984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 4199
I20260620 02:35:51.714165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:35:51.714843 388 external_mini_cluster.cc:1428] Setting key 088f6628ccb35dfd7cf483744e048ae0
Jun 20 02:35:51 dist-test-slave-rpg6 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:35:51.742417 4329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:51.716586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37433 (local address 127.0.97.62:37481)
0620 02:35:51.717128 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:51.717134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:51.718338 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 02:35:51.720408 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:51.720418 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:51.720522 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:51.720667 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:51.722845 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:51.724107 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:51.728202 (+ 4095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:51.728684 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:51.728808 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:51.734500 (+ 5692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:51.734510 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:51.734518 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:51.734582 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:51.738652 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:51.739888 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:51.739894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:51.739899 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:51.740041 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:51.740601 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:51.740605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:51.740608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:51.740915 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:51.741029 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:51.741913 (+ 884us) server_negotiation.cc:299] Negotiation successful
0620 02:35:51.742107 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":498,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:35:51.743772 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:51.717125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37481 (local address 127.0.97.2:37433)
0620 02:35:51.718117 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:51.718185 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:51.720062 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:51.720888 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:51.720898 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:51.721511 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:51.722520 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:51.722599 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:51.724701 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:51.724707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:51.727671 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:51.727705 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:51.728996 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:51.729008 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:51.729388 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:51.730310 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:51.730348 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:51.733795 (+ 3447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:51.738904 (+ 5109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:51.738915 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:51.738933 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:51.739678 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:51.740216 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:51.740220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:51.740223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:51.740450 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:51.741060 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:51.741067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:51.741509 (+ 442us) client_negotiation.cc:769] Sending connection context
0620 02:35:51.741794 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 02:35:51.742266 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":432,"threads_started":1}
I20260620 02:35:51.745803 4327 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37481
I20260620 02:35:51.746150 4327 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:51.747116 4327 heartbeater.cc:511] Master 127.0.97.62:37481 requested a full tablet report, sending...
I20260620 02:35:51.749243 4004 ts_manager.cc:194] Registered new tserver with Master: d998e601411b44b884686501586bf269 (127.0.97.2:45311)
I20260620 02:35:51.750130 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:35:51.773422 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37481
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:52.128427 4333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:52.128738 4333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:52.128834 4333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:52.136564 4333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:52.136691 4333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:52.136869 4333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:35:52.147225 4333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:52.148655 4333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:52.149971 4333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:52.164959 4338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:52.164832 4341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:52.164862 4339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:52.166550 4333 server_base.cc:1034] running on GCE node
I20260620 02:35:52.167318 4333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:52.168442 4333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:52.169775 4333 hybrid_clock.cc:648] HybridClock initialized: now 1781922952169687 us; error 62 us; skew 500 ppm
I20260620 02:35:52.170161 4333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:52 dist-test-slave-rpg6 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:52.173749 4333 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:35:52.175458 4333 webserver.cc:460] Webserver started at http://127.0.97.3:42223/ using document root <none> and password file <none>
I20260620 02:35:52.176326 4333 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:52.176488 4333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:52.176826 4333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:52.179467 4193 heartbeater.cc:503] Master 127.0.97.62:37481 was elected leader, sending a full tablet report...
I20260620 02:35:52.180202 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "f826a75d710046769f87d9f7bbbf64cc"
format_stamp: "Formatted at 2026-06-20 02:35:52 on dist-test-slave-rpg6"
server_key: "646edb4fb0ee17096a03bede79d07621"
server_key_iv: "439911b2d3351594188b8b7a30caa939"
server_key_version: "encryptionkey@0"
I20260620 02:35:52.181344 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f826a75d710046769f87d9f7bbbf64cc"
format_stamp: "Formatted at 2026-06-20 02:35:52 on dist-test-slave-rpg6"
server_key: "646edb4fb0ee17096a03bede79d07621"
server_key_iv: "439911b2d3351594188b8b7a30caa939"
server_key_version: "encryptionkey@0"
I20260620 02:35:52.188005 4333 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:35:52.192282 4348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:52.193969 4333 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:35:52.194298 4333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "f826a75d710046769f87d9f7bbbf64cc"
format_stamp: "Formatted at 2026-06-20 02:35:52 on dist-test-slave-rpg6"
server_key: "646edb4fb0ee17096a03bede79d07621"
server_key_iv: "439911b2d3351594188b8b7a30caa939"
server_key_version: "encryptionkey@0"
I20260620 02:35:52.194538 4333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:52.220875 4333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:52.226339 4333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:52.226686 4333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:52.227851 4333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:52.229348 4333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:52.229470 4333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:52.229622 4333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:52.229698 4333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:52.280314 4333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35351
I20260620 02:35:52.280344 4460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35351 every 8 connection(s)
I20260620 02:35:52.281649 4333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:35:52.288549 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 4333
I20260620 02:35:52.288733 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:35:52.289330 388 external_mini_cluster.cc:1428] Setting key 4e44f1659ac43d23402994f453fa5c0b
Jun 20 02:35:52 dist-test-slave-rpg6 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:52.307718 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:52.284447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57515 (local address 127.0.97.62:37481)
0620 02:35:52.284965 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:52.284970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:52.286297 (+ 1327us) server_negotiation.cc:407] Connection header received
0620 02:35:52.288740 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:52.288750 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:52.288870 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:52.289011 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:52.291512 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.292759 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.295051 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.295789 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.295953 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:52.300921 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:52.300928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:52.300931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:52.300971 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:52.303915 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.304813 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.304818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.304820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.304947 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.305464 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.305469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.305472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.305855 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:52.305992 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:52.307057 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0620 02:35:52.307299 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":306,"threads_started":1}
I20260620 02:35:52.308871 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:52.284804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37481 (local address 127.0.97.3:57515)
0620 02:35:52.285909 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:52.286059 (+ 150us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:52.288319 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:52.289359 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:52.289374 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:52.290285 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:52.291254 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:52.291288 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.292981 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.292989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:52.294805 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:52.294837 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.296300 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.296308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:52.296584 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:52.297476 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:52.297513 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:52.300537 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:52.304108 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:52.304116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:52.304133 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:52.304648 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:52.305077 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:52.305081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:52.305083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:52.305310 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:52.306067 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:52.306080 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:52.306560 (+ 480us) client_negotiation.cc:769] Sending connection context
0620 02:35:52.306939 (+ 379us) client_negotiation.cc:240] Negotiation successful
0620 02:35:52.307488 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":463,"threads_started":1}
I20260620 02:35:52.310945 4461 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37481
I20260620 02:35:52.311379 4461 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:52.312261 4461 heartbeater.cc:511] Master 127.0.97.62:37481 requested a full tablet report, sending...
I20260620 02:35:52.314081 4004 ts_manager.cc:194] Registered new tserver with Master: f826a75d710046769f87d9f7bbbf64cc (127.0.97.3:35351)
I20260620 02:35:52.314992 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57515
I20260620 02:35:52.321756 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:35:52 dist-test-slave-rpg6 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:35:52 dist-test-slave-rpg6 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:35:52.378649 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:52.360850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.0.97.1:41489)
0620 02:35:52.361531 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:52.361540 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:52.361574 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:35:52.361637 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:52.361643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:52.361860 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:52.361989 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:52.363705 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.364851 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.366254 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.366937 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.367046 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:52.371670 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:52.371684 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:52.371700 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:52.371745 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:52.374811 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.375730 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.375740 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.375743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.376169 (+ 426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.376691 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.376696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.376698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.377401 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:52.377660 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:52.378089 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 02:35:52.378321 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":399,"threads_started":1}
Jun 20 02:35:52 dist-test-slave-rpg6 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:52.401804 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:52.383992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.0.97.62:37481)
0620 02:35:52.384252 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:52.384260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:52.384432 (+ 172us) server_negotiation.cc:407] Connection header received
0620 02:35:52.384539 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:52.384543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:52.384609 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:52.384696 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:52.386040 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.386823 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.388211 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.388619 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.388721 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:52.394012 (+ 5291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:52.394020 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:52.394023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:52.394072 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:52.397702 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.398687 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.398692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.398694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.398863 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.399431 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.399440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.399442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.399910 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:52.400238 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:52.401025 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 02:35:52.401457 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 02:35:52.402475 4004 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47944
I20260620 02:35:52.404044 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 4065
I20260620 02:35:52.422528 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:52.405355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.0.97.1:41489)
0620 02:35:52.405545 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:52.405550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:52.405740 (+ 190us) server_negotiation.cc:407] Connection header received
0620 02:35:52.405814 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:52.405818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:52.405885 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:52.405980 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:52.408334 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.409303 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.411041 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.411535 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.411694 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:52.416258 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:52.416264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:52.416268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:52.416318 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:52.419367 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.420312 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.420320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.420323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.420498 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.420928 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.420939 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.420941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.421224 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:52.421521 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:52.421911 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 02:35:52.422164 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:35:52.423415 4188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:52.512593 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 4065
I20260620 02:35:52.529932 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 4199
Jun 20 02:35:52 dist-test-slave-rpg6 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:35:52.547688 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:52.530821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.0.97.2:45311)
0620 02:35:52.531311 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:52.531319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:52.531344 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:35:52.531410 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:52.531416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:52.531628 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:52.531818 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:52.533050 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.534146 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.535436 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.536072 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.536217 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:52.540957 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:52.540972 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:52.540991 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:52.541035 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:52.544372 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.545329 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.545337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.545340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.545482 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.545931 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.545936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.545941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.546329 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:52.546664 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:52.547002 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 02:35:52.547290 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":245,"threads_started":1}
I20260620 02:35:52.549146 4322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:52.634879 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 4199
I20260620 02:35:52.651651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 4333
Jun 20 02:35:52 dist-test-slave-rpg6 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:35:52.671026 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:52.652950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.0.97.3:35351)
0620 02:35:52.653812 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:52.653819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:52.653849 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:35:52.653911 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:52.653918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:52.654136 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:52.654336 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:52.655972 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.657047 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.658685 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:52.659440 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:52.659545 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:52.664725 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:52.664741 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:52.664766 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:52.664816 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:52.667718 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.668591 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.668599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.668601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.668738 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:52.669328 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:52.669332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:52.669334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:52.669704 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:52.669964 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:52.670374 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 02:35:52.670589 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":564,"threads_started":1}
I20260620 02:35:52.672601 4456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:52.759603 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 4333
I20260620 02:35:52.775908 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 3972
I20260620 02:35:52.777122 4034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:52.882664 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 3972
2026-06-20T02:35:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4311 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:35:52 dist-test-slave-rpg6 krb5kdc[4481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:35:52 dist-test-slave-rpg6 krb5kdc[4481](info): set up 2 sockets
Jun 20 02:35:52 dist-test-slave-rpg6 krb5kdc[4481](info): commencing operation
krb5kdc: starting...
W20260620 02:35:53.997517 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:35:54 dist-test-slave-rpg6 krb5kdc[4481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:35:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:35:54.225492 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43333
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:54.580087 4494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:54.580578 4494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:54.580824 4494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:54.588866 4494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:35:54.588996 4494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:54.589042 4494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:54.589092 4494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:35:54.589140 4494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:35:54.599551 4494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:54.601020 4494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:54.602336 4494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:54.615013 4502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:54.615056 4500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:54.615068 4499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:54.616557 4494 server_base.cc:1034] running on GCE node
I20260620 02:35:54.617529 4494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:54.619141 4494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:54.620370 4494 hybrid_clock.cc:648] HybridClock initialized: now 1781922954620288 us; error 68 us; skew 500 ppm
I20260620 02:35:54.620839 4494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:54 dist-test-slave-rpg6 krb5kdc[4481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:54.624310 4494 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:35:54.625694 4494 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:54.625803 4494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:54.626076 4494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:54.629701 4494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "7a9c6947b8fd4fc296cc58e16fd19b43"
format_stamp: "Formatted at 2026-06-20 02:35:54 on dist-test-slave-rpg6"
server_key: "9bbf862318130bf523c92cf342be81b3"
server_key_iv: "cadba581c128a7f1e8cb06101188dea3"
server_key_version: "encryptionkey@0"
I20260620 02:35:54.630554 4494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "7a9c6947b8fd4fc296cc58e16fd19b43"
format_stamp: "Formatted at 2026-06-20 02:35:54 on dist-test-slave-rpg6"
server_key: "9bbf862318130bf523c92cf342be81b3"
server_key_iv: "cadba581c128a7f1e8cb06101188dea3"
server_key_version: "encryptionkey@0"
I20260620 02:35:54.636502 4494 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 02:35:54.640836 4508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:54.642776 4494 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:35:54.643033 4494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "7a9c6947b8fd4fc296cc58e16fd19b43"
format_stamp: "Formatted at 2026-06-20 02:35:54 on dist-test-slave-rpg6"
server_key: "9bbf862318130bf523c92cf342be81b3"
server_key_iv: "cadba581c128a7f1e8cb06101188dea3"
server_key_version: "encryptionkey@0"
I20260620 02:35:54.643388 4494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:54.662691 4494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:54.667503 4494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:54.667848 4494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:54.688438 4494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43333
I20260620 02:35:54.688462 4559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43333 every 8 connection(s)
I20260620 02:35:54.689502 4494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:35:54.694456 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:54.699703 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 4494
I20260620 02:35:54.700256 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:35:54.701074 388 external_mini_cluster.cc:1428] Setting key b195ac09323921df09e306d96894ab99
I20260620 02:35:54.706647 4560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43: Bootstrap starting.
I20260620 02:35:54.710768 4560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:54.712299 4560 log.cc:824] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:54.716279 4560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43: No bootstrap required, opened a new log
Jun 20 02:35:54 dist-test-slave-rpg6 krb5kdc[4481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:54.724957 4560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9c6947b8fd4fc296cc58e16fd19b43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43333 } }
I20260620 02:35:54.725458 4560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:54.725567 4560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9c6947b8fd4fc296cc58e16fd19b43, State: Initialized, Role: FOLLOWER
I20260620 02:35:54.726254 4560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9c6947b8fd4fc296cc58e16fd19b43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43333 } }
I20260620 02:35:54.726526 4560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:35:54.726714 4560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:35:54.726925 4560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:54.728890 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:54.703395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.0.97.62:43333)
0620 02:35:54.704461 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:54.704478 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:54.704529 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:35:54.706353 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:54.706380 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:54.706900 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:54.707298 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:54.709427 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:54.710869 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:54.712869 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:54.714079 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:54.714235 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:54.719742 (+ 5507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:54.719756 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:54.719780 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:54.720039 (+ 259us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:54.724165 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:54.725052 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:54.725059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:54.725066 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:54.725253 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:54.725661 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:54.725666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:54.725671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:54.726379 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:54.726720 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:54.727209 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 02:35:54.727685 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":512,"threads_started":1}
I20260620 02:35:54.730012 4560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9c6947b8fd4fc296cc58e16fd19b43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43333 } }
I20260620 02:35:54.730681 4560 leader_election.cc:302] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a9c6947b8fd4fc296cc58e16fd19b43; no voters:
I20260620 02:35:54.731374 4560 leader_election.cc:288] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:35:54.731535 4565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:54.733179 4565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [term 1 LEADER]: Becoming Leader. State: Replica: 7a9c6947b8fd4fc296cc58e16fd19b43, State: Running, Role: LEADER
I20260620 02:35:54.733707 4560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:35:54.733681 4565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9c6947b8fd4fc296cc58e16fd19b43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43333 } }
I20260620 02:35:54.737661 4566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a9c6947b8fd4fc296cc58e16fd19b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9c6947b8fd4fc296cc58e16fd19b43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43333 } } }
I20260620 02:35:54.737649 4567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a9c6947b8fd4fc296cc58e16fd19b43. Latest consensus state: current_term: 1 leader_uuid: "7a9c6947b8fd4fc296cc58e16fd19b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9c6947b8fd4fc296cc58e16fd19b43" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43333 } } }
I20260620 02:35:54.738166 4566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:54.738201 4567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43 [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:54.739108 4575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:35:54.746203 4575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:35:54.757874 4575 catalog_manager.cc:1347] Generated new cluster ID: c744c0f231194024b77a7035c7b4b6f7
I20260620 02:35:54.758036 4575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:35:54.791589 4575 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:35:54.793061 4575 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:35:54.801357 4575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43: Generated new TSK 0
I20260620 02:35:54.802582 4575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:35:54.860978 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43333
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:55.217945 4586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:55.218264 4586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:55.218367 4586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:55.226471 4586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:55.226613 4586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:55.226734 4586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:35:55.237823 4586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:55.239249 4586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:55.240630 4586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:55.256143 4591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:55.256181 4592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:55.256206 4594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:55.257108 4586 server_base.cc:1034] running on GCE node
I20260620 02:35:55.257777 4586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:55.258858 4586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:55.260102 4586 hybrid_clock.cc:648] HybridClock initialized: now 1781922955260013 us; error 85 us; skew 500 ppm
I20260620 02:35:55.260495 4586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:55 dist-test-slave-rpg6 krb5kdc[4481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:55.264683 4586 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:35:55.266314 4586 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:55.266487 4586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:55.266870 4586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:55.270105 4586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "2f258b989eae4fef9746b34f99eb70f4"
format_stamp: "Formatted at 2026-06-20 02:35:55 on dist-test-slave-rpg6"
server_key: "00d94e58a66251ab15bdd8152aea6d81"
server_key_iv: "75279b1f30f1a4661bd4cb94836bf807"
server_key_version: "encryptionkey@0"
I20260620 02:35:55.271015 4586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2f258b989eae4fef9746b34f99eb70f4"
format_stamp: "Formatted at 2026-06-20 02:35:55 on dist-test-slave-rpg6"
server_key: "00d94e58a66251ab15bdd8152aea6d81"
server_key_iv: "75279b1f30f1a4661bd4cb94836bf807"
server_key_version: "encryptionkey@0"
I20260620 02:35:55.276904 4586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:35:55.281682 4600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:55.283193 4586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260620 02:35:55.283417 4586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "2f258b989eae4fef9746b34f99eb70f4"
format_stamp: "Formatted at 2026-06-20 02:35:55 on dist-test-slave-rpg6"
server_key: "00d94e58a66251ab15bdd8152aea6d81"
server_key_iv: "75279b1f30f1a4661bd4cb94836bf807"
server_key_version: "encryptionkey@0"
I20260620 02:35:55.283658 4586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:55.302043 4586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:55.306869 4586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:55.307174 4586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:55.308317 4586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:55.309876 4586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:55.310015 4586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:55.310148 4586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:55.310245 4586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:55.351020 4586 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41491
I20260620 02:35:55.351231 4712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41491 every 8 connection(s)
I20260620 02:35:55.352286 4586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:35:55.353366 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 4586
I20260620 02:35:55.353593 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:35:55.354241 388 external_mini_cluster.cc:1428] Setting key 2af364728c487b813f97f23f00c047ab
Jun 20 02:35:55 dist-test-slave-rpg6 krb5kdc[4481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:35:55.383236 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:55.355342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50619 (local address 127.0.97.62:43333)
0620 02:35:55.356326 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:55.356349 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:55.357196 (+ 847us) server_negotiation.cc:407] Connection header received
0620 02:35:55.359363 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:55.359373 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:55.359487 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:55.359635 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:55.361625 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:55.362814 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:55.365395 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:55.365820 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:55.365934 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:55.376595 (+ 10661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:55.376601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:55.376605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:55.376646 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:55.379357 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:55.380553 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:55.380558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:55.380560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:55.380681 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:55.381207 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:55.381210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:55.381212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:55.381491 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:55.381609 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:55.382753 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0620 02:35:55.382946 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":685,"threads_started":1}
I20260620 02:35:55.385015 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:55.355971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43333 (local address 127.0.97.1:50619)
0620 02:35:55.356967 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:55.357024 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:55.358929 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:55.360024 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:55.360034 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:55.360729 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:55.361380 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:55.361405 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:55.363023 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:55.363028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:55.365179 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:55.365196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:55.366071 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:55.366081 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:55.366447 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:55.372901 (+ 6454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:55.372954 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:55.376200 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:55.379634 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:55.379644 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:55.379661 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:55.380389 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:55.380805 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:55.380814 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:55.380820 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:55.381082 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:55.381685 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:55.381697 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:55.382228 (+ 531us) client_negotiation.cc:769] Sending connection context
0620 02:35:55.382580 (+ 352us) client_negotiation.cc:240] Negotiation successful
0620 02:35:55.383388 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":468,"threads_started":1}
I20260620 02:35:55.387306 4713 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43333
I20260620 02:35:55.387663 4713 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:55.388530 4713 heartbeater.cc:511] Master 127.0.97.62:43333 requested a full tablet report, sending...
I20260620 02:35:55.391105 4525 ts_manager.cc:194] Registered new tserver with Master: 2f258b989eae4fef9746b34f99eb70f4 (127.0.97.1:41491)
I20260620 02:35:55.393395 4525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:35:55.411576 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43333
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:55.766660 4719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:55.767024 4719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:55.767225 4719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:55.775005 4719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:55.775141 4719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:55.775345 4719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:35:55.786546 4719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:55.788230 4719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:55.789667 4719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:55.804569 4724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:55.804584 4727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:55.804620 4725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:55.805385 4719 server_base.cc:1034] running on GCE node
I20260620 02:35:55.806098 4719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:55.807281 4719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:55.808549 4719 hybrid_clock.cc:648] HybridClock initialized: now 1781922955808477 us; error 67 us; skew 500 ppm
I20260620 02:35:55.808954 4719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:55 dist-test-slave-rpg6 krb5kdc[4481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:55.812893 4719 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:35:55.814286 4719 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:55.814446 4719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:55.814813 4719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:55.818367 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "db9e715b25a4402c9c5a249d545256d8"
format_stamp: "Formatted at 2026-06-20 02:35:55 on dist-test-slave-rpg6"
server_key: "2765bc64c73e04d88c510ee09d020dd0"
server_key_iv: "9000d3b92c145cb179e6ed6ce5e84265"
server_key_version: "encryptionkey@0"
I20260620 02:35:55.819392 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "db9e715b25a4402c9c5a249d545256d8"
format_stamp: "Formatted at 2026-06-20 02:35:55 on dist-test-slave-rpg6"
server_key: "2765bc64c73e04d88c510ee09d020dd0"
server_key_iv: "9000d3b92c145cb179e6ed6ce5e84265"
server_key_version: "encryptionkey@0"
I20260620 02:35:55.825397 4719 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260620 02:35:55.829612 4733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:55.831382 4719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 02:35:55.831705 4719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "db9e715b25a4402c9c5a249d545256d8"
format_stamp: "Formatted at 2026-06-20 02:35:55 on dist-test-slave-rpg6"
server_key: "2765bc64c73e04d88c510ee09d020dd0"
server_key_iv: "9000d3b92c145cb179e6ed6ce5e84265"
server_key_version: "encryptionkey@0"
I20260620 02:35:55.832050 4719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:55.850361 4719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:55.855594 4719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:55.855997 4719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:55.857081 4719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:55.858639 4719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:55.858738 4719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:55.858880 4719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:55.858984 4719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:55.894078 4719 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45535
I20260620 02:35:55.894115 4845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45535 every 8 connection(s)
I20260620 02:35:55.895040 4719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:35:55.904112 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 4719
I20260620 02:35:55.904306 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:35:55.904763 388 external_mini_cluster.cc:1428] Setting key 0d4f964eed142ef2a67b24cab72827fa
Jun 20 02:35:55 dist-test-slave-rpg6 krb5kdc[4481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:55.918670 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:55.897854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36733 (local address 127.0.97.62:43333)
0620 02:35:55.898303 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:55.898308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:55.899252 (+ 944us) server_negotiation.cc:407] Connection header received
0620 02:35:55.901273 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:55.901279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:55.901386 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:55.901519 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:55.903243 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:55.904185 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:55.906272 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:55.906917 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:55.907059 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:55.911190 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:55.911196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:55.911199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:55.911236 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:55.914938 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:55.915992 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:55.915998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:55.916002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:55.916157 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:55.916660 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:55.916671 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:55.916673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:55.916963 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:55.917129 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:55.917914 (+ 785us) server_negotiation.cc:299] Negotiation successful
0620 02:35:55.918162 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":253,"threads_started":1}
I20260620 02:35:55.919785 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:55.898377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43333 (local address 127.0.97.2:36733)
0620 02:35:55.899076 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:55.899115 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:55.900948 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:55.901795 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:55.901805 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:55.902388 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:55.902985 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:55.903003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:55.904442 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:55.904451 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:55.906093 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:55.906108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:55.907185 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:55.907196 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:55.907439 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:55.908254 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:55.908284 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:55.910882 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:55.915204 (+ 4322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:55.915214 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:55.915233 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:55.915774 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:55.916299 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:55.916304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:55.916308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:55.916520 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:55.917109 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:55.917118 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:55.917511 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 02:35:55.917815 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 02:35:55.918557 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":275,"threads_started":1}
I20260620 02:35:55.921710 4846 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43333
I20260620 02:35:55.922106 4846 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:55.923099 4846 heartbeater.cc:511] Master 127.0.97.62:43333 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:35:55.925098 4525 ts_manager.cc:194] Registered new tserver with Master: db9e715b25a4402c9c5a249d545256d8 (127.0.97.2:45535)
I20260620 02:35:55.925977 4525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:35:55.955278 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43333
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:35:56.319095 4852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:56.319414 4852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:56.319514 4852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:56.327502 4852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:56.327651 4852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:56.327771 4852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:35:56.339406 4852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:56.340934 4852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:56.342329 4852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:56.356804 4857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:56.356853 4860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:56.356801 4858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:56.357896 4852 server_base.cc:1034] running on GCE node
I20260620 02:35:56.358639 4852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:56.359656 4852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:56.360889 4852 hybrid_clock.cc:648] HybridClock initialized: now 1781922956360838 us; error 63 us; skew 500 ppm
I20260620 02:35:56.361268 4852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:35:56 dist-test-slave-rpg6 krb5kdc[4481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:35:56.364837 4852 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:35:56.366185 4852 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:56.366285 4852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:56.366698 4852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:56.369861 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "a8b6710686264971968e3bef7df2b0de"
format_stamp: "Formatted at 2026-06-20 02:35:56 on dist-test-slave-rpg6"
server_key: "35b80dc2bddbbe965aa5b1b150b7209c"
server_key_iv: "ab45a613fb486c338ca4e7119f47445f"
server_key_version: "encryptionkey@0"
I20260620 02:35:56.370759 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a8b6710686264971968e3bef7df2b0de"
format_stamp: "Formatted at 2026-06-20 02:35:56 on dist-test-slave-rpg6"
server_key: "35b80dc2bddbbe965aa5b1b150b7209c"
server_key_iv: "ab45a613fb486c338ca4e7119f47445f"
server_key_version: "encryptionkey@0"
I20260620 02:35:56.376561 4852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 02:35:56.380929 4866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:56.382681 4852 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:35:56.382907 4852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "a8b6710686264971968e3bef7df2b0de"
format_stamp: "Formatted at 2026-06-20 02:35:56 on dist-test-slave-rpg6"
server_key: "35b80dc2bddbbe965aa5b1b150b7209c"
server_key_iv: "ab45a613fb486c338ca4e7119f47445f"
server_key_version: "encryptionkey@0"
I20260620 02:35:56.383196 4852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:56.397050 4713 heartbeater.cc:503] Master 127.0.97.62:43333 was elected leader, sending a full tablet report...
I20260620 02:35:56.409111 4852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:56.414458 4852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:56.414867 4852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:56.416045 4852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:56.417519 4852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:56.417620 4852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:56.417696 4852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:56.417752 4852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:56.456219 4852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46435
I20260620 02:35:56.456254 4978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46435 every 8 connection(s)
I20260620 02:35:56.457175 4852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:35:56.458755 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 4852
I20260620 02:35:56.458938 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:35:56.459450 388 external_mini_cluster.cc:1428] Setting key 1f9227e897f194bc708f9b9b7a9d0ab6
Jun 20 02:35:56 dist-test-slave-rpg6 krb5kdc[4481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:35:56.480285 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.459975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48869 (local address 127.0.97.62:43333)
0620 02:35:56.460639 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:56.460644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:56.461867 (+ 1223us) server_negotiation.cc:407] Connection header received
0620 02:35:56.463958 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:56.463967 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:56.464036 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:56.464136 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:56.465803 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.466627 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.468397 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.468874 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.468981 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.473195 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:56.473207 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:56.473212 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:56.473262 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:56.476595 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:56.477605 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:56.477610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:56.477613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:56.477756 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:56.478319 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:56.478324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:56.478326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:56.478658 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:56.478798 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:56.479623 (+ 825us) server_negotiation.cc:299] Negotiation successful
0620 02:35:56.479812 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":291,"threads_started":1}
I20260620 02:35:56.481200 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.460798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43333 (local address 127.0.97.3:48869)
0620 02:35:56.461700 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:56.461753 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:56.463615 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:56.464442 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:56.464451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:56.465029 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:56.465611 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.465629 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.466793 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.466796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.468249 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.468263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.469096 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.469101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.469371 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.470098 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:35:56.470127 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:56.472766 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:56.476827 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:56.476836 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:56.476853 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:56.477418 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:56.477906 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:35:56.477910 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:35:56.477913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:35:56.478170 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:35:56.478807 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:56.478814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:56.479207 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 02:35:56.479486 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 02:35:56.480010 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":413,"threads_started":1}
I20260620 02:35:56.482906 4979 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43333
I20260620 02:35:56.483264 4979 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:56.484071 4979 heartbeater.cc:511] Master 127.0.97.62:43333 requested a full tablet report, sending...
I20260620 02:35:56.485759 4525 ts_manager.cc:194] Registered new tserver with Master: a8b6710686264971968e3bef7df2b0de (127.0.97.3:46435)
I20260620 02:35:56.486790 4525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48869
I20260620 02:35:56.488857 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:35:56.507755 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.493779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.0.97.62:43333)
0620 02:35:56.493968 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:56.493973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:56.494497 (+ 524us) server_negotiation.cc:407] Connection header received
0620 02:35:56.494665 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:56.494669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:56.494736 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:56.494832 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:56.496512 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.497340 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.498753 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.499199 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.499311 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.501847 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:56.501853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:56.501856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:56.501907 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:56.504976 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:56.505833 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:56.505838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:56.505840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:56.505980 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:56.506423 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:56.506427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:56.506430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:56.506807 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:56.506993 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:56.507347 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 02:35:56.507524 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:35:56.512596 4524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:35:56.519944 4524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:35:56.554739 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.533419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41491 (local address 127.0.0.1:54110)
0620 02:35:56.544421 (+ 11002us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:56.544467 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:56.544724 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:56.546325 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:56.546337 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:56.546382 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:56.546902 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.546915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.549104 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.549110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.551587 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.551620 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.553719 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.553726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.553935 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.553987 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 02:35:56.554137 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 02:35:56.554350 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10697,"thread_start_us":10540,"threads_started":1}
I20260620 02:35:56.554729 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.534141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.0.97.1:41491)
0620 02:35:56.545060 (+ 10919us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:56.545071 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:56.545110 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:35:56.545224 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:56.545233 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:56.545518 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:56.545919 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:56.547118 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.548802 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.551861 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.553503 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.553664 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.553911 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:56.554202 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 02:35:56.554391 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10707,"thread_start_us":10554,"threads_started":1}
I20260620 02:35:56.556078 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.533947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45535 (local address 127.0.0.1:49126)
0620 02:35:56.545902 (+ 11955us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:56.545923 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:56.546063 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:56.547893 (+ 1830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:56.547899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:56.547935 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:56.548598 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.548612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.550714 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.550720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.553005 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.553020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.555122 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.555128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.555359 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.555439 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 02:35:56.555519 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 02:35:56.555657 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11803,"mutex_wait_us":67,"thread_start_us":11118,"threads_started":1}
I20260620 02:35:56.556133 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.533499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46435 (local address 127.0.0.1:58032)
0620 02:35:56.545327 (+ 11828us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:56.545347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:56.545472 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:56.547336 (+ 1864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:56.547342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:56.547369 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:56.548004 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.548153 (+ 149us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.550521 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.550526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.552941 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.552954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.555168 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.555176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.555398 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.555491 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 02:35:56.555574 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 02:35:56.555711 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11702,"mutex_wait_us":92,"thread_start_us":10210,"threads_started":1}
I20260620 02:35:56.556948 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.534402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.0.97.2:45535)
0620 02:35:56.546993 (+ 12591us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:56.547001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:56.547026 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:35:56.547121 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:56.547127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:56.547357 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:56.547552 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:56.548951 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.550507 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.553214 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.554646 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.554788 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.554982 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:56.556476 (+ 1494us) server_negotiation.cc:299] Negotiation successful
0620 02:35:56.556628 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12451,"thread_start_us":12343,"threads_started":1}
I20260620 02:35:56.558467 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.534285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.0.97.3:46435)
0620 02:35:56.546684 (+ 12399us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:56.546693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:56.546731 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:35:56.546859 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:56.546866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:56.547128 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:56.547329 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:56.548436 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.550351 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.553164 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.554636 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.554776 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.555017 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:56.558063 (+ 3046us) server_negotiation.cc:299] Negotiation successful
0620 02:35:56.558216 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12212,"thread_start_us":12029,"threads_started":1}
I20260620 02:35:56.559648 4648 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad4cf375bbc4e1ab2936c4324ae685a (DEFAULT_TABLE table=test-table [id=28eec0e699ba4e4db1920074df8cbe83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:56.561249 4648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad4cf375bbc4e1ab2936c4324ae685a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:56.560737 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad4cf375bbc4e1ab2936c4324ae685a (DEFAULT_TABLE table=test-table [id=28eec0e699ba4e4db1920074df8cbe83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:56.561619 4914 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad4cf375bbc4e1ab2936c4324ae685a (DEFAULT_TABLE table=test-table [id=28eec0e699ba4e4db1920074df8cbe83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:56.562394 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad4cf375bbc4e1ab2936c4324ae685a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:56.563261 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad4cf375bbc4e1ab2936c4324ae685a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:56.575541 4998 tablet_bootstrap.cc:492] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: Bootstrap starting.
I20260620 02:35:56.579783 4998 tablet_bootstrap.cc:654] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:56.580956 4998 log.cc:824] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:56.582417 4999 tablet_bootstrap.cc:492] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: Bootstrap starting.
I20260620 02:35:56.583914 5000 tablet_bootstrap.cc:492] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: Bootstrap starting.
I20260620 02:35:56.584637 4998 tablet_bootstrap.cc:492] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: No bootstrap required, opened a new log
I20260620 02:35:56.584939 4998 ts_tablet_manager.cc:1397] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 02:35:56.585983 4999 tablet_bootstrap.cc:654] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:56.588132 5000 tablet_bootstrap.cc:654] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:56.588732 4999 log.cc:824] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:56.589504 5000 log.cc:824] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:56.592232 4999 tablet_bootstrap.cc:492] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: No bootstrap required, opened a new log
I20260620 02:35:56.592561 4999 ts_tablet_manager.cc:1397] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 02:35:56.594662 5000 tablet_bootstrap.cc:492] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: No bootstrap required, opened a new log
I20260620 02:35:56.594985 5000 ts_tablet_manager.cc:1397] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260620 02:35:56.596903 4998 raft_consensus.cc:348] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f258b989eae4fef9746b34f99eb70f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41491 } } peers { permanent_uuid: "a8b6710686264971968e3bef7df2b0de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46435 } } peers { permanent_uuid: "db9e715b25a4402c9c5a249d545256d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45535 } }
I20260620 02:35:56.597564 4998 raft_consensus.cc:374] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:56.597728 4998 raft_consensus.cc:729] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b6710686264971968e3bef7df2b0de, State: Initialized, Role: FOLLOWER
I20260620 02:35:56.598457 4998 consensus_queue.cc:260] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f258b989eae4fef9746b34f99eb70f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41491 } } peers { permanent_uuid: "a8b6710686264971968e3bef7df2b0de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46435 } } peers { permanent_uuid: "db9e715b25a4402c9c5a249d545256d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45535 } }
I20260620 02:35:56.601404 4999 raft_consensus.cc:348] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f258b989eae4fef9746b34f99eb70f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41491 } } peers { permanent_uuid: "a8b6710686264971968e3bef7df2b0de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46435 } } peers { permanent_uuid: "db9e715b25a4402c9c5a249d545256d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45535 } }
I20260620 02:35:56.601837 4999 raft_consensus.cc:374] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:56.601999 4999 raft_consensus.cc:729] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f258b989eae4fef9746b34f99eb70f4, State: Initialized, Role: FOLLOWER
I20260620 02:35:56.602592 4979 heartbeater.cc:503] Master 127.0.97.62:43333 was elected leader, sending a full tablet report...
I20260620 02:35:56.602943 4999 consensus_queue.cc:260] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f258b989eae4fef9746b34f99eb70f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41491 } } peers { permanent_uuid: "a8b6710686264971968e3bef7df2b0de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46435 } } peers { permanent_uuid: "db9e715b25a4402c9c5a249d545256d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45535 } }
I20260620 02:35:56.603430 4998 ts_tablet_manager.cc:1428] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260620 02:35:56.605170 4999 ts_tablet_manager.cc:1428] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260620 02:35:56.605854 4714 tablet.cc:2369] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:35:56.606950 5000 raft_consensus.cc:348] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f258b989eae4fef9746b34f99eb70f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41491 } } peers { permanent_uuid: "a8b6710686264971968e3bef7df2b0de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46435 } } peers { permanent_uuid: "db9e715b25a4402c9c5a249d545256d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45535 } }
I20260620 02:35:56.607491 5000 raft_consensus.cc:374] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:56.607609 5000 raft_consensus.cc:729] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9e715b25a4402c9c5a249d545256d8, State: Initialized, Role: FOLLOWER
I20260620 02:35:56.608376 5000 consensus_queue.cc:260] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f258b989eae4fef9746b34f99eb70f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41491 } } peers { permanent_uuid: "a8b6710686264971968e3bef7df2b0de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46435 } } peers { permanent_uuid: "db9e715b25a4402c9c5a249d545256d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45535 } }
I20260620 02:35:56.609931 4846 heartbeater.cc:503] Master 127.0.97.62:43333 was elected leader, sending a full tablet report...
I20260620 02:35:56.611575 5000 ts_tablet_manager.cc:1428] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260620 02:35:56.647446 4847 tablet.cc:2369] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:35:56.670147 5005 raft_consensus.cc:482] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:35:56.670531 5005 raft_consensus.cc:504] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f258b989eae4fef9746b34f99eb70f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41491 } } peers { permanent_uuid: "a8b6710686264971968e3bef7df2b0de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46435 } } peers { permanent_uuid: "db9e715b25a4402c9c5a249d545256d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45535 } }
I20260620 02:35:56.673166 5005 leader_election.cc:288] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8b6710686264971968e3bef7df2b0de (127.0.97.3:46435), db9e715b25a4402c9c5a249d545256d8 (127.0.97.2:45535)
I20260620 02:35:56.682864 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.674004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52871 (local address 127.0.97.3:46435)
0620 02:35:56.674228 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:56.674237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:56.674505 (+ 268us) server_negotiation.cc:407] Connection header received
0620 02:35:56.674665 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:56.674671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:56.674901 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:56.675026 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:56.675780 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.677111 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.680469 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.681741 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.681897 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.682019 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:56.682537 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 02:35:56.682663 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 02:35:56.683331 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.673554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46435 (local address 127.0.97.1:52871)
0620 02:35:56.674381 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:56.674400 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:56.674544 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:56.675060 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:56.675065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:56.675123 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:56.675596 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.675608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.677287 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.677295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.680199 (+ 2904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.680244 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.682054 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.682062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.682308 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.682367 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 02:35:56.682512 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 02:35:56.682716 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":86,"thread_start_us":649,"threads_started":1}
I20260620 02:35:56.684325 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.674342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57101 (local address 127.0.97.2:45535)
0620 02:35:56.674570 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:56.674577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:56.674599 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:35:56.674681 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:56.674687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:56.674748 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:56.674859 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:35:56.675869 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.677150 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.680849 (+ 3699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.682192 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.682352 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.682466 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:56.683921 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0620 02:35:56.684069 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:35:56.684396 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.673516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45535 (local address 127.0.97.1:57101)
0620 02:35:56.674280 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:56.674303 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:56.674482 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:56.674889 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:56.674894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:56.674929 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:35:56.675598 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.675618 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.677387 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.677395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.680628 (+ 3233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:35:56.680651 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.682451 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.682458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:35:56.683690 (+ 1232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.683719 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 02:35:56.683842 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 02:35:56.684017 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":60,"thread_start_us":1486,"threads_started":1}
I20260620 02:35:56.684619 4934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad4cf375bbc4e1ab2936c4324ae685a" candidate_uuid: "2f258b989eae4fef9746b34f99eb70f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b6710686264971968e3bef7df2b0de" is_pre_election: true
I20260620 02:35:56.685315 4934 raft_consensus.cc:2454] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f258b989eae4fef9746b34f99eb70f4 in term 0.
I20260620 02:35:56.685568 4801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad4cf375bbc4e1ab2936c4324ae685a" candidate_uuid: "2f258b989eae4fef9746b34f99eb70f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9e715b25a4402c9c5a249d545256d8" is_pre_election: true
I20260620 02:35:56.686103 4801 raft_consensus.cc:2454] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f258b989eae4fef9746b34f99eb70f4 in term 0.
I20260620 02:35:56.686259 4603 leader_election.cc:302] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f258b989eae4fef9746b34f99eb70f4, a8b6710686264971968e3bef7df2b0de; no voters:
I20260620 02:35:56.686853 5005 raft_consensus.cc:2781] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:35:56.687033 5005 raft_consensus.cc:482] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:35:56.687258 5005 raft_consensus.cc:3037] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:56.691483 5005 raft_consensus.cc:504] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f258b989eae4fef9746b34f99eb70f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41491 } } peers { permanent_uuid: "a8b6710686264971968e3bef7df2b0de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46435 } } peers { permanent_uuid: "db9e715b25a4402c9c5a249d545256d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45535 } }
I20260620 02:35:56.692952 5005 leader_election.cc:288] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [CANDIDATE]: Term 1 election: Requested vote from peers a8b6710686264971968e3bef7df2b0de (127.0.97.3:46435), db9e715b25a4402c9c5a249d545256d8 (127.0.97.2:45535)
I20260620 02:35:56.693302 4801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad4cf375bbc4e1ab2936c4324ae685a" candidate_uuid: "2f258b989eae4fef9746b34f99eb70f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db9e715b25a4402c9c5a249d545256d8"
I20260620 02:35:56.693331 4934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad4cf375bbc4e1ab2936c4324ae685a" candidate_uuid: "2f258b989eae4fef9746b34f99eb70f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8b6710686264971968e3bef7df2b0de"
I20260620 02:35:56.693588 4801 raft_consensus.cc:3037] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:56.693619 4934 raft_consensus.cc:3037] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:56.697130 4801 raft_consensus.cc:2454] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f258b989eae4fef9746b34f99eb70f4 in term 1.
I20260620 02:35:56.697320 4934 raft_consensus.cc:2454] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f258b989eae4fef9746b34f99eb70f4 in term 1.
I20260620 02:35:56.697750 4604 leader_election.cc:302] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f258b989eae4fef9746b34f99eb70f4, db9e715b25a4402c9c5a249d545256d8; no voters:
I20260620 02:35:56.698065 5005 raft_consensus.cc:2781] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:56.698637 5005 raft_consensus.cc:686] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f258b989eae4fef9746b34f99eb70f4, State: Running, Role: LEADER
I20260620 02:35:56.699237 5005 consensus_queue.cc:237] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f258b989eae4fef9746b34f99eb70f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41491 } } peers { permanent_uuid: "a8b6710686264971968e3bef7df2b0de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46435 } } peers { permanent_uuid: "db9e715b25a4402c9c5a249d545256d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45535 } }
I20260620 02:35:56.704869 4524 catalog_manager.cc:5495] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f258b989eae4fef9746b34f99eb70f4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2f258b989eae4fef9746b34f99eb70f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f258b989eae4fef9746b34f99eb70f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8b6710686264971968e3bef7df2b0de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db9e715b25a4402c9c5a249d545256d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45535 } health_report { overall_health: UNKNOWN } } }
W20260620 02:35:56.708896 4980 tablet.cc:2369] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:35:56.754705 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.748785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54122 (local address 127.0.97.1:41491)
0620 02:35:56.748995 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:56.749003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:56.749068 (+ 65us) server_negotiation.cc:407] Connection header received
0620 02:35:56.749397 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:56.749403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:56.749492 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:56.749656 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:35:56.750111 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.750981 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.752446 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.752857 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.752967 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.753511 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:35:56.754041 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:35:56.754170 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:56.754387 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 02:35:56.754459 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:35:56.763455 4801 raft_consensus.cc:1261] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8 [term 1 FOLLOWER]: Refusing update from remote peer 2f258b989eae4fef9746b34f99eb70f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:56.763482 4934 raft_consensus.cc:1261] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de [term 1 FOLLOWER]: Refusing update from remote peer 2f258b989eae4fef9746b34f99eb70f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:56.764554 5009 consensus_queue.cc:1035] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db9e715b25a4402c9c5a249d545256d8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:56.764936 5005 consensus_queue.cc:1035] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8b6710686264971968e3bef7df2b0de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:56.778481 5014 mvcc.cc:205] Tried to move back new op lower bound from 7298756430892011520 to 7298756430644244480. Current Snapshot: MvccSnapshot[applied={T|T < 7298756430892011520}]
I20260620 02:35:56.790745 4523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:56.791039 4523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:56.795121 4523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a9c6947b8fd4fc296cc58e16fd19b43: Sending DeleteTablet for 3 replicas of tablet 2ad4cf375bbc4e1ab2936c4324ae685a
I20260620 02:35:56.796494 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad4cf375bbc4e1ab2936c4324ae685a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49126
I20260620 02:35:56.796499 4648 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad4cf375bbc4e1ab2936c4324ae685a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54110
I20260620 02:35:56.797283 4914 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad4cf375bbc4e1ab2936c4324ae685a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58032
I20260620 02:35:56.797499 5021 tablet_replica.cc:330] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: stopping tablet replica
I20260620 02:35:56.797977 5021 raft_consensus.cc:2229] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:56.798528 5021 raft_consensus.cc:2258] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:56.799391 5022 tablet_replica.cc:330] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: stopping tablet replica
I20260620 02:35:56.799600 5023 tablet_replica.cc:330] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: stopping tablet replica
I20260620 02:35:56.799935 5022 raft_consensus.cc:2229] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:35:56.800061 5023 raft_consensus.cc:2229] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:56.800630 5023 raft_consensus.cc:2258] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:56.800668 5022 raft_consensus.cc:2258] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:56.801862 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 4586
I20260620 02:35:56.802974 5021 ts_tablet_manager.cc:1905] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:56.802983 5022 ts_tablet_manager.cc:1905] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:56.810364 5023 ts_tablet_manager.cc:1905] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:56.810745 5021 ts_tablet_manager.cc:1918] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:35:56.810993 5021 log.cc:1196] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/2ad4cf375bbc4e1ab2936c4324ae685a
I20260620 02:35:56.811652 5021 ts_tablet_manager.cc:1939] T 2ad4cf375bbc4e1ab2936c4324ae685a P db9e715b25a4402c9c5a249d545256d8: Deleting consensus metadata
I20260620 02:35:56.813721 4512 catalog_manager.cc:4841] TS db9e715b25a4402c9c5a249d545256d8 (127.0.97.2:45535): tablet 2ad4cf375bbc4e1ab2936c4324ae685a (table test-table [id=28eec0e699ba4e4db1920074df8cbe83]) successfully deleted
I20260620 02:35:56.817688 5023 ts_tablet_manager.cc:1918] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:35:56.817857 5022 ts_tablet_manager.cc:1918] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:35:56.817979 5023 log.cc:1196] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/2ad4cf375bbc4e1ab2936c4324ae685a
I20260620 02:35:56.818073 5022 log.cc:1196] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/2ad4cf375bbc4e1ab2936c4324ae685a
I20260620 02:35:56.818578 5023 ts_tablet_manager.cc:1939] T 2ad4cf375bbc4e1ab2936c4324ae685a P a8b6710686264971968e3bef7df2b0de: Deleting consensus metadata
I20260620 02:35:56.818641 5022 ts_tablet_manager.cc:1939] T 2ad4cf375bbc4e1ab2936c4324ae685a P 2f258b989eae4fef9746b34f99eb70f4: Deleting consensus metadata
Jun 20 02:35:56 dist-test-slave-rpg6 krb5kdc[4481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:35:56.820463 4510 catalog_manager.cc:4841] TS 2f258b989eae4fef9746b34f99eb70f4 (127.0.97.1:41491): tablet 2ad4cf375bbc4e1ab2936c4324ae685a (table test-table [id=28eec0e699ba4e4db1920074df8cbe83]) successfully deleted
I20260620 02:35:56.820598 4511 catalog_manager.cc:4841] TS a8b6710686264971968e3bef7df2b0de (127.0.97.3:46435): tablet 2ad4cf375bbc4e1ab2936c4324ae685a (table test-table [id=28eec0e699ba4e4db1920074df8cbe83]) successfully deleted
I20260620 02:35:56.828415 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.803334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.0.97.1:41491)
0620 02:35:56.803496 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:56.803505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:56.808345 (+ 4840us) server_negotiation.cc:407] Connection header received
0620 02:35:56.808617 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:56.808623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:56.808718 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:56.808813 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:56.810529 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.811676 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.814323 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.815041 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.815165 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.820655 (+ 5490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:56.820670 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:56.820690 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:56.820740 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:56.824871 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:56.825812 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:56.825823 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:56.825826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:56.826009 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:56.826586 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:56.826592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:56.826595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:56.827083 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:56.827469 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:56.827709 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 02:35:56.827991 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 02:35:56.829298 4708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:56.937366 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 4586
I20260620 02:35:56.955632 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 4719
Jun 20 02:35:56 dist-test-slave-rpg6 krb5kdc[4481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:35:56.972738 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:56.956535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.0.97.2:45535)
0620 02:35:56.956704 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:56.956708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:56.956727 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:35:56.956799 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:56.956803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:56.956875 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:56.956964 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:56.958281 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.959041 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.960663 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:56.961104 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:56.961234 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:56.965802 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:56.965819 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:56.965836 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:56.965883 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:56.969718 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:56.970516 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:56.970524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:56.970526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:56.970666 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:56.971079 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:56.971083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:56.971085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:56.971490 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:56.971713 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:56.972213 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 02:35:56.972447 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:35:56.973652 4841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:57.068451 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 4719
I20260620 02:35:57.087393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 4852
Jun 20 02:35:57 dist-test-slave-rpg6 krb5kdc[4481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:35:57.105747 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:57.088357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.0.97.3:46435)
0620 02:35:57.088571 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:57.088578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:57.088726 (+ 148us) server_negotiation.cc:407] Connection header received
0620 02:35:57.088982 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:57.088986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:57.089059 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:57.089283 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:57.090634 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:57.091383 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:57.092956 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:35:57.093382 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:35:57.093531 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:35:57.098751 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:57.098769 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:57.098786 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:35:57.098835 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:57.102469 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:57.103386 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:57.103395 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:57.103397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:57.103548 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:35:57.104049 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:35:57.104055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:35:57.104058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:35:57.104497 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:57.104818 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:57.105147 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 02:35:57.105401 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 02:35:57.106401 4974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:57.198050 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 4852
I20260620 02:35:57.215533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 4494
I20260620 02:35:57.216954 4555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:57.330734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 4494
2026-06-20T02:35:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4448 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T02:35:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:35:57Z Disabled control of system clock
I20260620 02:35:57.402169 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45471
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 02:35:57.760677 5032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:57.761015 5032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:57.761116 5032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:57.768950 5032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:35:57.769217 5032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:57.769265 5032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:57.769318 5032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:35:57.769357 5032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:35:57.779690 5032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45471
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:57.781085 5032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:57.783253 5032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:57.797398 5040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:57.797372 5037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:57.797461 5038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:57.798728 5032 server_base.cc:1034] running on GCE node
I20260620 02:35:57.799643 5032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:57.801301 5032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:57.802565 5032 hybrid_clock.cc:648] HybridClock initialized: now 1781922957802490 us; error 69 us; skew 500 ppm
I20260620 02:35:57.803042 5032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:35:57.806399 5032 webserver.cc:460] Webserver started at http://127.0.97.62:44925/ using document root <none> and password file <none>
I20260620 02:35:57.807255 5032 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:57.807363 5032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:57.807730 5032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:57.810956 5032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "c67602b8f84d41ec910461429f42c2db"
format_stamp: "Formatted at 2026-06-20 02:35:57 on dist-test-slave-rpg6"
server_key: "a5c34a548a6fc4a764e5a00262aa91b3"
server_key_iv: "07393c9aae300f2060dcd7f93af9ae65"
server_key_version: "encryptionkey@0"
I20260620 02:35:57.811949 5032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "c67602b8f84d41ec910461429f42c2db"
format_stamp: "Formatted at 2026-06-20 02:35:57 on dist-test-slave-rpg6"
server_key: "a5c34a548a6fc4a764e5a00262aa91b3"
server_key_iv: "07393c9aae300f2060dcd7f93af9ae65"
server_key_version: "encryptionkey@0"
I20260620 02:35:57.818917 5032 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260620 02:35:57.823489 5046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:57.825188 5032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:35:57.825414 5032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "c67602b8f84d41ec910461429f42c2db"
format_stamp: "Formatted at 2026-06-20 02:35:57 on dist-test-slave-rpg6"
server_key: "a5c34a548a6fc4a764e5a00262aa91b3"
server_key_iv: "07393c9aae300f2060dcd7f93af9ae65"
server_key_version: "encryptionkey@0"
I20260620 02:35:57.825675 5032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:57.832118 5032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:57.832962 5032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:57.833276 5032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:57.853662 5032 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45471
I20260620 02:35:57.853678 5097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45471 every 8 connection(s)
I20260620 02:35:57.855149 5032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:35:57.859192 5098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:57.865162 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 5032
I20260620 02:35:57.865478 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:35:57.866132 388 external_mini_cluster.cc:1428] Setting key 8fe9607ea045ee8d4ecf8a284880bb99
I20260620 02:35:57.870983 5098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db: Bootstrap starting.
I20260620 02:35:57.874845 5098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:57.875530 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:57.868570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.0.97.62:45471)
0620 02:35:57.869673 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:57.869686 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:57.869729 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:35:57.871548 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:57.871575 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:57.872105 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:57.872563 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:57.873713 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:57.873719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:57.873732 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:57.873745 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:57.873807 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:57.873824 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:57.874042 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:57.874190 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 02:35:57.874455 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":366,"threads_started":1}
I20260620 02:35:57.876523 5098 log.cc:824] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:57.880523 5098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db: No bootstrap required, opened a new log
I20260620 02:35:57.888296 5098 raft_consensus.cc:348] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67602b8f84d41ec910461429f42c2db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45471 } }
I20260620 02:35:57.888723 5098 raft_consensus.cc:374] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:57.888819 5098 raft_consensus.cc:729] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67602b8f84d41ec910461429f42c2db, State: Initialized, Role: FOLLOWER
I20260620 02:35:57.889394 5098 consensus_queue.cc:260] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67602b8f84d41ec910461429f42c2db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45471 } }
I20260620 02:35:57.889679 5098 raft_consensus.cc:388] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:35:57.889860 5098 raft_consensus.cc:482] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:35:57.890081 5098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:57.892657 5098 raft_consensus.cc:504] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67602b8f84d41ec910461429f42c2db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45471 } }
I20260620 02:35:57.893355 5098 leader_election.cc:302] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67602b8f84d41ec910461429f42c2db; no voters:
I20260620 02:35:57.894054 5098 leader_election.cc:288] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:35:57.894331 5103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:57.895159 5103 raft_consensus.cc:686] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [term 1 LEADER]: Becoming Leader. State: Replica: c67602b8f84d41ec910461429f42c2db, State: Running, Role: LEADER
I20260620 02:35:57.895911 5103 consensus_queue.cc:237] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67602b8f84d41ec910461429f42c2db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45471 } }
I20260620 02:35:57.896651 5098 sys_catalog.cc:564] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:35:57.900902 5105 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [sys.catalog]: SysCatalogTable state changed. Reason: New leader c67602b8f84d41ec910461429f42c2db. Latest consensus state: current_term: 1 leader_uuid: "c67602b8f84d41ec910461429f42c2db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67602b8f84d41ec910461429f42c2db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45471 } } }
I20260620 02:35:57.901791 5105 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:57.901321 5104 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c67602b8f84d41ec910461429f42c2db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67602b8f84d41ec910461429f42c2db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45471 } } }
I20260620 02:35:57.902385 5104 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db [sys.catalog]: This master's current role is: LEADER
I20260620 02:35:57.903440 5113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:35:57.909358 5113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:35:57.919775 5113 catalog_manager.cc:1347] Generated new cluster ID: 0a9196b5b73a49debb4e362b4e54f646
I20260620 02:35:57.919965 5113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:35:57.964197 5113 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:35:57.964785 5113 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:35:57.974567 5113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db: Generated new TSK 0
I20260620 02:35:57.975783 5113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:35:57.983445 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45471
--builtin_ntp_servers=127.0.97.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 02:35:58.349031 5122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:58.349355 5122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:58.349509 5122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:58.357602 5122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:58.357733 5122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:58.357923 5122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:35:58.367666 5122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:58.369130 5122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:58.370942 5122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:58.385672 5128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:58.385725 5130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:58.385645 5127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:58.386631 5122 server_base.cc:1034] running on GCE node
I20260620 02:35:58.387269 5122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:58.388420 5122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:58.389659 5122 hybrid_clock.cc:648] HybridClock initialized: now 1781922958389618 us; error 42 us; skew 500 ppm
I20260620 02:35:58.390023 5122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:35:58.392915 5122 webserver.cc:460] Webserver started at http://127.0.97.1:45711/ using document root <none> and password file <none>
I20260620 02:35:58.393688 5122 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:58.393836 5122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:58.394176 5122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:58.397576 5122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "29c272061a3b420abc9504f75d93f709"
format_stamp: "Formatted at 2026-06-20 02:35:58 on dist-test-slave-rpg6"
server_key: "123d1296d3392abcec004a0229c71aa0"
server_key_iv: "8084a39babe05e9f2c6eedf0b9ddaa84"
server_key_version: "encryptionkey@0"
I20260620 02:35:58.398554 5122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "29c272061a3b420abc9504f75d93f709"
format_stamp: "Formatted at 2026-06-20 02:35:58 on dist-test-slave-rpg6"
server_key: "123d1296d3392abcec004a0229c71aa0"
server_key_iv: "8084a39babe05e9f2c6eedf0b9ddaa84"
server_key_version: "encryptionkey@0"
I20260620 02:35:58.404361 5122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 02:35:58.408416 5136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:58.410614 5122 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:35:58.410843 5122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "29c272061a3b420abc9504f75d93f709"
format_stamp: "Formatted at 2026-06-20 02:35:58 on dist-test-slave-rpg6"
server_key: "123d1296d3392abcec004a0229c71aa0"
server_key_iv: "8084a39babe05e9f2c6eedf0b9ddaa84"
server_key_version: "encryptionkey@0"
I20260620 02:35:58.411118 5122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:58.417531 5122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:58.418480 5122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:58.418840 5122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:58.419950 5122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:58.421495 5122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:58.421626 5122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:58.421767 5122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:58.421855 5122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:58.457864 5122 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42899
I20260620 02:35:58.457912 5248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42899 every 8 connection(s)
I20260620 02:35:58.459306 5122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:35:58.466408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 5122
I20260620 02:35:58.466651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:35:58.467470 388 external_mini_cluster.cc:1428] Setting key 381738bcf9130096c62a602803ed308a
I20260620 02:35:58.468796 5251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:58.462206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45293 (local address 127.0.97.62:45471)
0620 02:35:58.462641 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:58.462646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:58.463974 (+ 1328us) server_negotiation.cc:407] Connection header received
0620 02:35:58.465778 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:58.465782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:58.465850 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:58.465973 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:58.467812 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:58.467817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:58.467874 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:58.467901 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:58.467957 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:58.467972 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:58.468080 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:58.468374 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 02:35:58.468448 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":225,"threads_started":1}
I20260620 02:35:58.470019 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:58.462825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45471 (local address 127.0.97.1:45293)
0620 02:35:58.463747 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:58.463791 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:58.465453 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:58.466356 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:58.466372 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:58.466521 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:58.467375 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:35:58.467379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:58.467440 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:35:58.467578 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:58.468083 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:58.468096 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:58.468104 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 02:35:58.468401 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 02:35:58.468726 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":260,"threads_started":1}
I20260620 02:35:58.470716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45471
--builtin_ntp_servers=127.0.97.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 02:35:58.471871 5249 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45471
I20260620 02:35:58.472203 5249 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:58.472841 5249 heartbeater.cc:511] Master 127.0.97.62:45471 requested a full tablet report, sending...
I20260620 02:35:58.475221 5063 ts_manager.cc:194] Registered new tserver with Master: 29c272061a3b420abc9504f75d93f709 (127.0.97.1:42899)
W20260620 02:35:58.823658 5253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:58.824013 5253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:58.824106 5253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:58.832093 5253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:58.832206 5253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:58.832317 5253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:35:58.842130 5253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:58.843428 5253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:58.845427 5253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:58.860220 5258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:58.860251 5259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:58.861068 5261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:58.861155 5253 server_base.cc:1034] running on GCE node
I20260620 02:35:58.861910 5253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:58.863047 5253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:58.864418 5253 hybrid_clock.cc:648] HybridClock initialized: now 1781922958864314 us; error 51 us; skew 500 ppm
I20260620 02:35:58.865151 5253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:35:58.868446 5253 webserver.cc:460] Webserver started at http://127.0.97.2:42255/ using document root <none> and password file <none>
I20260620 02:35:58.869292 5253 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:58.869400 5253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:58.869800 5253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:58.872992 5253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "7f6e9e25703a41858cffefb23898ae7d"
format_stamp: "Formatted at 2026-06-20 02:35:58 on dist-test-slave-rpg6"
server_key: "391ec2cc83b3e7932a71e5676120250e"
server_key_iv: "331a59b898231da03b37596ce4328b63"
server_key_version: "encryptionkey@0"
I20260620 02:35:58.873862 5253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7f6e9e25703a41858cffefb23898ae7d"
format_stamp: "Formatted at 2026-06-20 02:35:58 on dist-test-slave-rpg6"
server_key: "391ec2cc83b3e7932a71e5676120250e"
server_key_iv: "331a59b898231da03b37596ce4328b63"
server_key_version: "encryptionkey@0"
I20260620 02:35:58.879938 5253 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:35:58.884182 5267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:58.885905 5253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:35:58.886137 5253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "7f6e9e25703a41858cffefb23898ae7d"
format_stamp: "Formatted at 2026-06-20 02:35:58 on dist-test-slave-rpg6"
server_key: "391ec2cc83b3e7932a71e5676120250e"
server_key_iv: "331a59b898231da03b37596ce4328b63"
server_key_version: "encryptionkey@0"
I20260620 02:35:58.886431 5253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:58.894490 5253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:58.895479 5253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:58.895800 5253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:58.896917 5253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:58.898595 5253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:58.898687 5253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:58.898864 5253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:58.898938 5253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:58.935750 5253 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36205
I20260620 02:35:58.935806 5379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36205 every 8 connection(s)
I20260620 02:35:58.937178 5253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:35:58.943970 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 5253
I20260620 02:35:58.944180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:35:58.944613 388 external_mini_cluster.cc:1428] Setting key 1334e8e6a999cdb9005bcf4d4b0a0f24
I20260620 02:35:58.945883 5251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:58.939932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49161 (local address 127.0.97.62:45471)
0620 02:35:58.940178 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:58.940183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:58.941538 (+ 1355us) server_negotiation.cc:407] Connection header received
0620 02:35:58.943251 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:58.943257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:58.943319 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:58.943433 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:58.945079 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:58.945084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:58.945089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:58.945109 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:58.945151 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:58.945169 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:58.945290 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:58.945529 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 02:35:58.945600 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:35:58.947086 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:58.940417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45471 (local address 127.0.97.2:49161)
0620 02:35:58.941379 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:58.941421 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:58.942958 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:58.943673 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:58.943682 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:58.943782 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:58.944735 (+ 953us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:35:58.944739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:58.944792 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:35:58.944892 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:58.945285 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:58.945295 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:58.945303 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 02:35:58.945571 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 02:35:58.945840 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":261,"threads_started":1}
I20260620 02:35:58.948355 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45471
--builtin_ntp_servers=127.0.97.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 02:35:58.948676 5380 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45471
I20260620 02:35:58.948998 5380 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:58.949595 5380 heartbeater.cc:511] Master 127.0.97.62:45471 requested a full tablet report, sending...
I20260620 02:35:58.951184 5063 ts_manager.cc:194] Registered new tserver with Master: 7f6e9e25703a41858cffefb23898ae7d (127.0.97.2:36205)
W20260620 02:35:59.312496 5383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:35:59.312887 5383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:35:59.313064 5383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:35:59.321113 5383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:35:59.321308 5383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:35:59.321499 5383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:35:59.332450 5383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:35:59.333896 5383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:35:59.335892 5383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:35:59.351642 5388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:59.351717 5389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:35:59.351696 5391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:35:59.352903 5383 server_base.cc:1034] running on GCE node
I20260620 02:35:59.353566 5383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:35:59.354581 5383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:35:59.355927 5383 hybrid_clock.cc:648] HybridClock initialized: now 1781922959355853 us; error 73 us; skew 500 ppm
I20260620 02:35:59.356359 5383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:35:59.359118 5383 webserver.cc:460] Webserver started at http://127.0.97.3:38661/ using document root <none> and password file <none>
I20260620 02:35:59.359952 5383 fs_manager.cc:359] Metadata directory not provided
I20260620 02:35:59.360057 5383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:35:59.360343 5383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:35:59.363559 5383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "e74a29873be544329387d13a41964253"
format_stamp: "Formatted at 2026-06-20 02:35:59 on dist-test-slave-rpg6"
server_key: "d2d90c8b446624bbe3cca2ece6280799"
server_key_iv: "6d7a899a64332f806933a06b249b4112"
server_key_version: "encryptionkey@0"
I20260620 02:35:59.364539 5383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e74a29873be544329387d13a41964253"
format_stamp: "Formatted at 2026-06-20 02:35:59 on dist-test-slave-rpg6"
server_key: "d2d90c8b446624bbe3cca2ece6280799"
server_key_iv: "6d7a899a64332f806933a06b249b4112"
server_key_version: "encryptionkey@0"
I20260620 02:35:59.370329 5383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:35:59.374398 5397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:59.376369 5383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:35:59.376600 5383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "e74a29873be544329387d13a41964253"
format_stamp: "Formatted at 2026-06-20 02:35:59 on dist-test-slave-rpg6"
server_key: "d2d90c8b446624bbe3cca2ece6280799"
server_key_iv: "6d7a899a64332f806933a06b249b4112"
server_key_version: "encryptionkey@0"
I20260620 02:35:59.376847 5383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:35:59.383325 5383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:35:59.384389 5383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:35:59.384683 5383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:35:59.385797 5383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:35:59.387414 5383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:35:59.387504 5383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:59.387666 5383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:35:59.387739 5383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:35:59.425688 5383 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38993
I20260620 02:35:59.425951 5509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38993 every 8 connection(s)
I20260620 02:35:59.427289 5383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:35:59.432060 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 5383
I20260620 02:35:59.432258 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:35:59.432736 388 external_mini_cluster.cc:1428] Setting key f8f326a16e4c0e91c9e688c6cc022db3
I20260620 02:35:59.437000 5251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.430504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40349 (local address 127.0.97.62:45471)
0620 02:35:59.430699 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:59.430704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:59.432532 (+ 1828us) server_negotiation.cc:407] Connection header received
0620 02:35:59.434495 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:59.434501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:59.434571 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:59.434704 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:59.436240 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:59.436246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:59.436251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:59.436270 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:59.436312 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:59.436324 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:59.436460 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:59.436664 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 02:35:59.436733 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:35:59.438094 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.431106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45471 (local address 127.0.97.3:40349)
0620 02:35:59.432343 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:59.432405 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:59.434195 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:59.434948 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:59.434957 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:59.435064 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:59.435923 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:35:59.435925 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:59.435967 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:35:59.436061 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:59.436430 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:59.436442 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:59.436450 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 02:35:59.436749 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 02:35:59.437035 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":356,"threads_started":1}
I20260620 02:35:59.440153 5510 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45471
I20260620 02:35:59.440649 5510 heartbeater.cc:464] Registering TS with master...
I20260620 02:35:59.441361 5510 heartbeater.cc:511] Master 127.0.97.62:45471 requested a full tablet report, sending...
I20260620 02:35:59.443017 5063 ts_manager.cc:194] Registered new tserver with Master: e74a29873be544329387d13a41964253 (127.0.97.3:38993)
I20260620 02:35:59.449115 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:35:59.456656 5251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.453822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49642 (local address 127.0.97.62:45471)
0620 02:35:59.454032 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:59.454040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:59.454519 (+ 479us) server_negotiation.cc:407] Connection header received
0620 02:35:59.454772 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:59.454776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:59.454831 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:59.454944 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:59.455977 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:59.455982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:59.455994 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:59.456012 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:59.456043 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:59.456055 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:59.456301 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:59.456355 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 02:35:59.456413 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:35:59.461067 5063 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:35:59.468037 5063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:35:59.478845 5249 heartbeater.cc:503] Master 127.0.97.62:45471 was elected leader, sending a full tablet report...
I20260620 02:35:59.498785 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.482642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38993 (local address 127.0.0.1:44642)
0620 02:35:59.483471 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:59.483516 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:59.483648 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:59.496079 (+ 12431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:59.496088 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:59.496195 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:59.497253 (+ 1058us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:35:59.497257 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:59.497290 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:35:59.497398 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:59.497927 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:59.497939 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:59.497946 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:35:59.498175 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 02:35:59.498292 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":414,"threads_started":1}
I20260620 02:35:59.498769 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.482721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.0.97.3:38993)
0620 02:35:59.495446 (+ 12725us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:59.495454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:59.495488 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:35:59.495554 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:59.495561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:59.495779 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:59.496049 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:59.497643 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:59.497656 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:59.497677 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:59.497699 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:59.497766 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:59.497798 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:59.498020 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:59.498219 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 02:35:59.498322 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12544,"thread_start_us":12366,"threads_started":1}
I20260620 02:35:59.500761 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.483124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42899 (local address 127.0.0.1:58028)
0620 02:35:59.490588 (+ 7464us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:59.490620 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:59.490763 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:59.497057 (+ 6294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:59.497062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:59.497087 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:59.497973 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:35:59.497976 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:59.497993 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:35:59.498039 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:59.500147 (+ 2108us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:59.500154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:59.500161 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:35:59.500293 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 02:35:59.500348 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":788,"threads_started":1}
I20260620 02:35:59.501384 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.484263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36205 (local address 127.0.0.1:35488)
0620 02:35:59.494478 (+ 10215us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:59.494501 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:59.494633 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:59.498756 (+ 4123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:59.498760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:59.498786 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:59.500246 (+ 1460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:35:59.500251 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:59.500275 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:35:59.500318 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:59.500762 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:59.500767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:59.500774 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:35:59.500924 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 02:35:59.500978 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10067,"thread_start_us":10460,"threads_started":1}
I20260620 02:35:59.501736 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.483609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.0.97.1:42899)
0620 02:35:59.496420 (+ 12811us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:59.496434 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:59.496495 (+ 61us) server_negotiation.cc:407] Connection header received
0620 02:35:59.496584 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:59.496595 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:59.496876 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:59.497058 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:59.499891 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:59.499900 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:59.499915 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:59.499935 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:59.500001 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:59.500025 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:59.501178 (+ 1153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:59.501252 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 02:35:59.501365 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12527,"thread_start_us":14382,"threads_started":1}
I20260620 02:35:59.502034 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.484578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.0.97.2:36205)
0620 02:35:59.498357 (+ 13779us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:59.498364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:59.498385 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:35:59.498438 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:59.498443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:59.498583 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:59.498762 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:59.500490 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:59.500499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:59.500512 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:59.500543 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:59.500602 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:59.500622 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:59.500778 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:59.501621 (+ 843us) server_negotiation.cc:299] Negotiation successful
0620 02:35:59.501721 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13627,"thread_start_us":13056,"threads_started":1}
I20260620 02:35:59.504172 5445 tablet_service.cc:1467] Processing CreateTablet for tablet 9bffc227999f48a8ac6fea36a83406b7 (DEFAULT_TABLE table=test-table [id=bddc14bcbfb6469585a9ded1ef316391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:59.505903 5445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bffc227999f48a8ac6fea36a83406b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:59.505493 5184 tablet_service.cc:1467] Processing CreateTablet for tablet 9bffc227999f48a8ac6fea36a83406b7 (DEFAULT_TABLE table=test-table [id=bddc14bcbfb6469585a9ded1ef316391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:59.506922 5184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bffc227999f48a8ac6fea36a83406b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:59.506405 5315 tablet_service.cc:1467] Processing CreateTablet for tablet 9bffc227999f48a8ac6fea36a83406b7 (DEFAULT_TABLE table=test-table [id=bddc14bcbfb6469585a9ded1ef316391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:35:59.507745 5315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bffc227999f48a8ac6fea36a83406b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:35:59.519891 5528 tablet_bootstrap.cc:492] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253: Bootstrap starting.
I20260620 02:35:59.522970 5529 tablet_bootstrap.cc:492] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d: Bootstrap starting.
I20260620 02:35:59.524255 5528 tablet_bootstrap.cc:654] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:59.524420 5530 tablet_bootstrap.cc:492] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709: Bootstrap starting.
I20260620 02:35:59.526307 5528 log.cc:824] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:59.527367 5529 tablet_bootstrap.cc:654] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:59.527582 5530 tablet_bootstrap.cc:654] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709: Neither blocks nor log segments found. Creating new log.
I20260620 02:35:59.528883 5530 log.cc:824] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:59.528889 5529 log.cc:824] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:35:59.530350 5528 tablet_bootstrap.cc:492] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253: No bootstrap required, opened a new log
I20260620 02:35:59.530782 5528 ts_tablet_manager.cc:1397] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260620 02:35:59.534566 5529 tablet_bootstrap.cc:492] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d: No bootstrap required, opened a new log
I20260620 02:35:59.534896 5529 ts_tablet_manager.cc:1397] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260620 02:35:59.534896 5530 tablet_bootstrap.cc:492] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709: No bootstrap required, opened a new log
I20260620 02:35:59.535199 5530 ts_tablet_manager.cc:1397] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260620 02:35:59.543030 5529 raft_consensus.cc:348] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74a29873be544329387d13a41964253" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "7f6e9e25703a41858cffefb23898ae7d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36205 } } peers { permanent_uuid: "29c272061a3b420abc9504f75d93f709" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } }
I20260620 02:35:59.543500 5529 raft_consensus.cc:374] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:59.543283 5528 raft_consensus.cc:348] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74a29873be544329387d13a41964253" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "7f6e9e25703a41858cffefb23898ae7d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36205 } } peers { permanent_uuid: "29c272061a3b420abc9504f75d93f709" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } }
I20260620 02:35:59.543614 5529 raft_consensus.cc:729] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6e9e25703a41858cffefb23898ae7d, State: Initialized, Role: FOLLOWER
I20260620 02:35:59.543794 5528 raft_consensus.cc:374] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:59.543941 5528 raft_consensus.cc:729] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74a29873be544329387d13a41964253, State: Initialized, Role: FOLLOWER
I20260620 02:35:59.544445 5529 consensus_queue.cc:260] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74a29873be544329387d13a41964253" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "7f6e9e25703a41858cffefb23898ae7d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36205 } } peers { permanent_uuid: "29c272061a3b420abc9504f75d93f709" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } }
I20260620 02:35:59.544837 5528 consensus_queue.cc:260] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74a29873be544329387d13a41964253" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "7f6e9e25703a41858cffefb23898ae7d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36205 } } peers { permanent_uuid: "29c272061a3b420abc9504f75d93f709" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } }
I20260620 02:35:59.545703 5380 heartbeater.cc:503] Master 127.0.97.62:45471 was elected leader, sending a full tablet report...
I20260620 02:35:59.546406 5529 ts_tablet_manager.cc:1428] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260620 02:35:59.547458 5530 raft_consensus.cc:348] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74a29873be544329387d13a41964253" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "7f6e9e25703a41858cffefb23898ae7d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36205 } } peers { permanent_uuid: "29c272061a3b420abc9504f75d93f709" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } }
I20260620 02:35:59.547922 5530 raft_consensus.cc:374] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:35:59.548049 5530 raft_consensus.cc:729] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c272061a3b420abc9504f75d93f709, State: Initialized, Role: FOLLOWER
I20260620 02:35:59.548738 5530 consensus_queue.cc:260] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74a29873be544329387d13a41964253" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "7f6e9e25703a41858cffefb23898ae7d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36205 } } peers { permanent_uuid: "29c272061a3b420abc9504f75d93f709" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } }
I20260620 02:35:59.550598 5530 ts_tablet_manager.cc:1428] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260620 02:35:59.551822 5510 heartbeater.cc:503] Master 127.0.97.62:45471 was elected leader, sending a full tablet report...
I20260620 02:35:59.552606 5528 ts_tablet_manager.cc:1428] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253: Time spent starting tablet: real 0.022s user 0.013s sys 0.009s
I20260620 02:35:59.597584 5535 raft_consensus.cc:482] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:35:59.597922 5535 raft_consensus.cc:504] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74a29873be544329387d13a41964253" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "7f6e9e25703a41858cffefb23898ae7d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36205 } } peers { permanent_uuid: "29c272061a3b420abc9504f75d93f709" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } }
I20260620 02:35:59.599714 5535 leader_election.cc:288] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6e9e25703a41858cffefb23898ae7d (127.0.97.2:36205), 29c272061a3b420abc9504f75d93f709 (127.0.97.1:42899)
I20260620 02:35:59.602624 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.600100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36205 (local address 127.0.97.3:36569)
0620 02:35:59.600273 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:59.600294 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:59.600444 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:59.600846 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:59.600853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:59.600886 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:59.601700 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:35:59.601703 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:59.601719 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:35:59.601760 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:59.602115 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:59.602120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:59.602125 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:35:59.602249 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 02:35:59.602308 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 02:35:59.602620 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.600177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36569 (local address 127.0.97.2:36205)
0620 02:35:59.600387 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:59.600394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:59.600420 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:35:59.600614 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:59.600619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:59.600671 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:59.600812 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:59.601923 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:59.601930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:59.601936 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:59.601964 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:59.601997 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:59.602011 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:59.602148 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:59.602249 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 02:35:59.602309 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 02:35:59.603331 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.600332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40965 (local address 127.0.97.1:42899)
0620 02:35:59.600561 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:59.600566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:59.601420 (+ 854us) server_negotiation.cc:407] Connection header received
0620 02:35:59.601583 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:59.601589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:59.601647 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:59.601753 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:59.602684 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:59.602692 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:59.602696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:59.602716 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:59.602761 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:59.602780 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:59.602895 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:59.603060 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 02:35:59.603127 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:35:59.603617 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.600103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42899 (local address 127.0.97.3:40965)
0620 02:35:59.601307 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:59.601321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:35:59.601448 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:35:59.601776 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:35:59.601781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:35:59.601802 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:35:59.602478 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:35:59.602480 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:35:59.602495 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:35:59.602529 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:35:59.602915 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:35:59.602920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:35:59.602926 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 02:35:59.603066 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 02:35:59.603122 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"mutex_wait_us":447,"thread_start_us":358,"threads_started":1}
I20260620 02:35:59.603920 5335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bffc227999f48a8ac6fea36a83406b7" candidate_uuid: "e74a29873be544329387d13a41964253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6e9e25703a41858cffefb23898ae7d" is_pre_election: true
I20260620 02:35:59.604511 5335 raft_consensus.cc:2454] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74a29873be544329387d13a41964253 in term 0.
I20260620 02:35:59.604727 5204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bffc227999f48a8ac6fea36a83406b7" candidate_uuid: "e74a29873be544329387d13a41964253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c272061a3b420abc9504f75d93f709" is_pre_election: true
I20260620 02:35:59.605324 5204 raft_consensus.cc:2454] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74a29873be544329387d13a41964253 in term 0.
I20260620 02:35:59.605682 5400 leader_election.cc:302] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6e9e25703a41858cffefb23898ae7d, e74a29873be544329387d13a41964253; no voters:
I20260620 02:35:59.606175 5535 raft_consensus.cc:2781] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:35:59.606336 5535 raft_consensus.cc:482] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:35:59.606443 5535 raft_consensus.cc:3037] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:59.608822 5535 raft_consensus.cc:504] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74a29873be544329387d13a41964253" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "7f6e9e25703a41858cffefb23898ae7d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36205 } } peers { permanent_uuid: "29c272061a3b420abc9504f75d93f709" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } }
I20260620 02:35:59.609522 5535 leader_election.cc:288] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [CANDIDATE]: Term 1 election: Requested vote from peers 7f6e9e25703a41858cffefb23898ae7d (127.0.97.2:36205), 29c272061a3b420abc9504f75d93f709 (127.0.97.1:42899)
I20260620 02:35:59.610060 5335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bffc227999f48a8ac6fea36a83406b7" candidate_uuid: "e74a29873be544329387d13a41964253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6e9e25703a41858cffefb23898ae7d"
I20260620 02:35:59.610097 5204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bffc227999f48a8ac6fea36a83406b7" candidate_uuid: "e74a29873be544329387d13a41964253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c272061a3b420abc9504f75d93f709"
I20260620 02:35:59.610324 5335 raft_consensus.cc:3037] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:59.610337 5204 raft_consensus.cc:3037] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:35:59.613745 5335 raft_consensus.cc:2454] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74a29873be544329387d13a41964253 in term 1.
I20260620 02:35:59.613752 5204 raft_consensus.cc:2454] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74a29873be544329387d13a41964253 in term 1.
I20260620 02:35:59.614301 5398 leader_election.cc:302] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c272061a3b420abc9504f75d93f709, e74a29873be544329387d13a41964253; no voters:
I20260620 02:35:59.614754 5535 raft_consensus.cc:2781] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:35:59.615332 5535 raft_consensus.cc:686] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 1 LEADER]: Becoming Leader. State: Replica: e74a29873be544329387d13a41964253, State: Running, Role: LEADER
I20260620 02:35:59.615998 5535 consensus_queue.cc:237] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74a29873be544329387d13a41964253" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } } peers { permanent_uuid: "7f6e9e25703a41858cffefb23898ae7d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36205 } } peers { permanent_uuid: "29c272061a3b420abc9504f75d93f709" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } }
I20260620 02:35:59.620911 5063 catalog_manager.cc:5495] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 reported cstate change: term changed from 0 to 1, leader changed from <none> to e74a29873be544329387d13a41964253 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e74a29873be544329387d13a41964253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74a29873be544329387d13a41964253" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6e9e25703a41858cffefb23898ae7d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29c272061a3b420abc9504f75d93f709" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } health_report { overall_health: UNKNOWN } } }
I20260620 02:35:59.654232 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.651944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.0.97.3:38993)
0620 02:35:59.652167 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:59.652174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:59.652332 (+ 158us) server_negotiation.cc:407] Connection header received
0620 02:35:59.652585 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:59.652590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:59.652671 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:59.652785 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:59.653632 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:59.653637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:59.653644 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:59.653663 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:59.653694 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:59.653708 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:59.653840 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:59.653917 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 02:35:59.653978 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:35:59.663174 5335 raft_consensus.cc:1261] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d [term 1 FOLLOWER]: Refusing update from remote peer e74a29873be544329387d13a41964253: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:59.663175 5204 raft_consensus.cc:1261] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709 [term 1 FOLLOWER]: Refusing update from remote peer e74a29873be544329387d13a41964253: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:35:59.664256 5538 consensus_queue.cc:1035] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c272061a3b420abc9504f75d93f709" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:59.664665 5535 consensus_queue.cc:1035] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6e9e25703a41858cffefb23898ae7d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:35:59.676467 5543 mvcc.cc:205] Tried to move back new op lower bound from 7298756442768818176 to 7298756442591313920. Current Snapshot: MvccSnapshot[applied={T|T < 7298756442768818176}]
I20260620 02:35:59.677486 5541 mvcc.cc:205] Tried to move back new op lower bound from 7298756442768818176 to 7298756442591313920. Current Snapshot: MvccSnapshot[applied={T|T < 7298756442768818176}]
I20260620 02:35:59.682919 5542 mvcc.cc:205] Tried to move back new op lower bound from 7298756442768818176 to 7298756442591313920. Current Snapshot: MvccSnapshot[applied={T|T < 7298756442768818176}]
I20260620 02:35:59.695511 5062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:59.695832 5062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:35:59.700019 5062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c67602b8f84d41ec910461429f42c2db: Sending DeleteTablet for 3 replicas of tablet 9bffc227999f48a8ac6fea36a83406b7
I20260620 02:35:59.701506 5315 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bffc227999f48a8ac6fea36a83406b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:59 UTC) from {username='slave'} at 127.0.0.1:35488
I20260620 02:35:59.701506 5184 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bffc227999f48a8ac6fea36a83406b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:59 UTC) from {username='slave'} at 127.0.0.1:58028
I20260620 02:35:59.701611 5445 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bffc227999f48a8ac6fea36a83406b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:35:59 UTC) from {username='slave'} at 127.0.0.1:44642
I20260620 02:35:59.704336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 5122
I20260620 02:35:59.715894 5550 tablet_replica.cc:330] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d: stopping tablet replica
I20260620 02:35:59.716427 5550 raft_consensus.cc:2229] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:59.716995 5550 raft_consensus.cc:2258] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:59.717605 5551 tablet_replica.cc:330] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709: stopping tablet replica
I20260620 02:35:59.718035 5551 raft_consensus.cc:2229] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:35:59.718537 5551 raft_consensus.cc:2258] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:59.721894 5550 ts_tablet_manager.cc:1905] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:59.726230 5552 tablet_replica.cc:330] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253: stopping tablet replica
I20260620 02:35:59.726877 5552 raft_consensus.cc:2229] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:35:59.727316 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.706436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.0.97.1:42899)
0620 02:35:59.706697 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:59.706707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:59.724425 (+ 17718us) server_negotiation.cc:407] Connection header received
0620 02:35:59.724651 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:59.724658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:59.724793 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:59.724930 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:59.726417 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:59.726424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:59.726428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:59.726452 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:59.726502 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:59.726519 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:59.726626 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:59.726923 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 02:35:59.726999 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 02:35:59.728353 5552 raft_consensus.cc:2258] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:35:59.728433 5244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:59.731362 5550 ts_tablet_manager.cc:1918] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:35:59.731594 5550 log.cc:1196] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/9bffc227999f48a8ac6fea36a83406b7
I20260620 02:35:59.731916 5552 ts_tablet_manager.cc:1905] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:59.732183 5550 ts_tablet_manager.cc:1939] T 9bffc227999f48a8ac6fea36a83406b7 P 7f6e9e25703a41858cffefb23898ae7d: Deleting consensus metadata
I20260620 02:35:59.734272 5049 catalog_manager.cc:4841] TS 7f6e9e25703a41858cffefb23898ae7d (127.0.97.2:36205): tablet 9bffc227999f48a8ac6fea36a83406b7 (table test-table [id=bddc14bcbfb6469585a9ded1ef316391]) successfully deleted
I20260620 02:35:59.741227 5552 ts_tablet_manager.cc:1918] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:35:59.741487 5552 log.cc:1196] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/9bffc227999f48a8ac6fea36a83406b7
I20260620 02:35:59.741986 5552 ts_tablet_manager.cc:1939] T 9bffc227999f48a8ac6fea36a83406b7 P e74a29873be544329387d13a41964253: Deleting consensus metadata
I20260620 02:35:59.743705 5049 catalog_manager.cc:4841] TS e74a29873be544329387d13a41964253 (127.0.97.3:38993): tablet 9bffc227999f48a8ac6fea36a83406b7 (table test-table [id=bddc14bcbfb6469585a9ded1ef316391]) successfully deleted
I20260620 02:35:59.730357 5551 ts_tablet_manager.cc:1905] T 9bffc227999f48a8ac6fea36a83406b7 P 29c272061a3b420abc9504f75d93f709: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:35:59.857157 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 5122
W20260620 02:35:59.881340 5047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42899 (error 108)
I20260620 02:35:59.881677 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 5253
I20260620 02:35:59.883661 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.882808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42899 (local address 127.0.0.1:58052)
0620 02:35:59.883068 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:59.883424 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260620 02:35:59.884155 5047 catalog_manager.cc:4584] TS 29c272061a3b420abc9504f75d93f709 (127.0.97.1:42899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9bffc227999f48a8ac6fea36a83406b7: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42899: connect: Connection refused (error 111)
I20260620 02:35:59.885990 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.882792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.0.97.2:36205)
0620 02:35:59.883007 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:59.883016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:59.883140 (+ 124us) server_negotiation.cc:407] Connection header received
0620 02:35:59.883428 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:59.883435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:59.883568 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:59.883703 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:59.885206 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:59.885214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:59.885219 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:59.885245 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:59.885289 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:59.885312 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:59.885474 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:59.885582 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 02:35:59.885650 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:35:59.886781 5375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:35:59.938840 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.938210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42899 (local address 127.0.0.1:58058)
0620 02:35:59.938485 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:59.938624 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 02:35:59.940181 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.939733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42899 (local address 127.0.0.1:58062)
0620 02:35:59.939930 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 02:35:59.940020 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 02:35:59.978771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 5253
I20260620 02:35:59.996336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 5383
I20260620 02:35:59.999970 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:35:59.997359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.0.97.3:38993)
0620 02:35:59.997565 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:35:59.997570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:35:59.997594 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:35:59.997846 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:35:59.997852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:35:59.997923 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:35:59.998066 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:35:59.999104 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:35:59.999111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:35:59.999118 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:35:59.999143 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:35:59.999192 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:35:59.999209 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:35:59.999387 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 02:35:59.999442 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 02:35:59.999505 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:36:00.000891 5505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:00.001592 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:00.000496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42899 (local address 127.0.0.1:58066)
0620 02:36:00.000711 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:00.000788 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 02:36:00.004170 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:00.003241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42899 (local address 127.0.0.1:58074)
0620 02:36:00.003540 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:00.003682 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 02:36:00.085125 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:00.084600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42899 (local address 127.0.0.1:58078)
0620 02:36:00.084845 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:00.084949 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 02:36:00.086359 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:00.085885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42899 (local address 127.0.0.1:58088)
0620 02:36:00.086073 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:00.086160 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 02:36:00.125190 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 5383
I20260620 02:36:00.145102 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 5032
I20260620 02:36:00.146386 5093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:00.254981 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 5032
2026-06-20T02:36:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2928 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 02:36:00.306627 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:36:00.310603 388 webserver.cc:460] Webserver started at https://127.0.0.1:44669/ using document root <none> and password file <none>
I20260620 02:36:00.311026 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:36:00.311898 388 webserver.cc:460] Webserver started at http://127.0.0.1:37809/ using document root <none> and password file <none>
2026-06-20T02:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:00Z Disabled control of system clock
I20260620 02:36:00.345141 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35451
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35451
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44669/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/testchainca.cert with env {}
W20260620 02:36:00.697849 5564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:00.698210 5564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:00.698304 5564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:00.706157 5564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:00.706290 5564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:00.706327 5564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:00.706377 5564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:00.706425 5564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:36:00.706487 5564 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:36:00.706528 5564 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44669/jwks.json
I20260620 02:36:00.717165 5564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44669/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:00.718542 5564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:00.719918 5564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:00.732900 5569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:00.732798 5570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:00.732872 5572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:00.733820 5564 server_base.cc:1034] running on GCE node
I20260620 02:36:00.734799 5564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:00.736485 5564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:00.737780 5564 hybrid_clock.cc:648] HybridClock initialized: now 1781922960737731 us; error 47 us; skew 500 ppm
I20260620 02:36:00.738283 5564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:00.747128 5564 webserver.cc:460] Webserver started at http://127.0.97.62:39269/ using document root <none> and password file <none>
I20260620 02:36:00.748067 5564 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:00.748237 5564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:00.748586 5564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:00.751773 5564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "fbf98aca7ffc434c943180ad709ee204"
format_stamp: "Formatted at 2026-06-20 02:36:00 on dist-test-slave-rpg6"
server_key: "165238a8937a86e3d9c07679c83744b6"
server_key_iv: "17f95490ad1e8e269aa36afcd487f25c"
server_key_version: "encryptionkey@0"
I20260620 02:36:00.752789 5564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "fbf98aca7ffc434c943180ad709ee204"
format_stamp: "Formatted at 2026-06-20 02:36:00 on dist-test-slave-rpg6"
server_key: "165238a8937a86e3d9c07679c83744b6"
server_key_iv: "17f95490ad1e8e269aa36afcd487f25c"
server_key_version: "encryptionkey@0"
I20260620 02:36:00.758638 5564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:36:00.762949 5578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:00.764739 5564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:36:00.764981 5564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "fbf98aca7ffc434c943180ad709ee204"
format_stamp: "Formatted at 2026-06-20 02:36:00 on dist-test-slave-rpg6"
server_key: "165238a8937a86e3d9c07679c83744b6"
server_key_iv: "17f95490ad1e8e269aa36afcd487f25c"
server_key_version: "encryptionkey@0"
I20260620 02:36:00.765209 5564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:00.797972 5564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:00.798925 5564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:00.799237 5564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:00.819659 5564 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35451
I20260620 02:36:00.819684 5629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35451 every 8 connection(s)
I20260620 02:36:00.821205 5564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:00.825400 5630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:00.828521 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 5564
I20260620 02:36:00.828750 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:36:00.829430 388 external_mini_cluster.cc:1428] Setting key 3c781282b950acc9f3ea5c53e21d6e9c
I20260620 02:36:00.840138 5630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204: Bootstrap starting.
I20260620 02:36:00.844560 5630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:00.845988 5630 log.cc:824] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:00.846854 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:00.831559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.0.97.62:35451)
0620 02:36:00.832782 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:00.832801 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:00.832863 (+ 62us) server_negotiation.cc:407] Connection header received
0620 02:36:00.835137 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:00.835180 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:00.835964 (+ 784us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:00.836497 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:00.837165 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:00.839167 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:00.840905 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:00.842018 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:00.842250 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:00.843563 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:00.843575 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:00.843595 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:00.844376 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:00.844519 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:00.844556 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:00.844683 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:00.845005 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 02:36:00.845396 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":426,"threads_started":1}
I20260620 02:36:00.849460 5630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204: No bootstrap required, opened a new log
I20260620 02:36:00.851184 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:36:00.858563 5630 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf98aca7ffc434c943180ad709ee204" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } }
I20260620 02:36:00.858973 5630 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:00.859055 5630 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf98aca7ffc434c943180ad709ee204, State: Initialized, Role: FOLLOWER
I20260620 02:36:00.859808 5630 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf98aca7ffc434c943180ad709ee204" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } }
I20260620 02:36:00.860178 5630 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:00.860345 5630 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:00.860649 5630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:00.863013 5630 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf98aca7ffc434c943180ad709ee204" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } }
I20260620 02:36:00.863812 5630 leader_election.cc:302] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbf98aca7ffc434c943180ad709ee204; no voters:
I20260620 02:36:00.864496 5630 leader_election.cc:288] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:36:00.864647 5635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:00.865514 5635 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [term 1 LEADER]: Becoming Leader. State: Replica: fbf98aca7ffc434c943180ad709ee204, State: Running, Role: LEADER
I20260620 02:36:00.866158 5635 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf98aca7ffc434c943180ad709ee204" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } }
I20260620 02:36:00.866968 5630 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:00.869858 5637 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf98aca7ffc434c943180ad709ee204. Latest consensus state: current_term: 1 leader_uuid: "fbf98aca7ffc434c943180ad709ee204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf98aca7ffc434c943180ad709ee204" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } } }
I20260620 02:36:00.870352 5637 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:00.870043 5636 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbf98aca7ffc434c943180ad709ee204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf98aca7ffc434c943180ad709ee204" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35451 } } }
I20260620 02:36:00.870515 5636 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:00.874764 5641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:00.880499 5641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:00.891173 5641 catalog_manager.cc:1347] Generated new cluster ID: 67e3177254d94893a6eba09e203b4a39
I20260620 02:36:00.891319 5641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:00.911414 5641 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:36:00.913435 5641 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:00.925655 5641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbf98aca7ffc434c943180ad709ee204: Generated new TSK 0
I20260620 02:36:00.926883 5641 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 02:36:00.972354 5660 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 02:36:00.972569 5660 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:36:00.973752 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:00.949910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.0.97.62:35451)
0620 02:36:00.950119 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:00.950126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:00.950888 (+ 762us) server_negotiation.cc:407] Connection header received
0620 02:36:00.951288 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:00.951296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:00.951409 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:00.951581 (+ 172us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:36:00.952243 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:00.953478 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:00.955060 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:00.955766 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:00.955948 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:00.956592 (+ 644us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:36:00.973001 (+ 16409us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 02:36:00.973410 (+ 409us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39034: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 02:36:00.973886 5633 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39034: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 02:36:00.975565 5659 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:00.949870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35451 (local address 127.0.0.1:39034)
0620 02:36:00.950751 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:00.950769 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:00.950955 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:00.951644 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:00.951647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:00.951659 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:36:00.952070 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:00.952080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:00.953666 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:00.953670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:00.954901 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:00.954914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:00.955954 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:00.956028 (+ 74us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:00.956183 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:00.956257 (+ 74us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:36:00.973336 (+ 17079us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 02:36:00.973639 (+ 303us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35451: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":508,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:35451: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44669/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 02:36:00.977763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 5564
I20260620 02:36:00.978996 5625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:01.077505 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 5564
2026-06-20T02:36:01Z chronyd exiting
I20260620 02:36:01.129077 388 test_util.cc:175] -----------------------------------------------
I20260620 02:36:01.129233 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781922915737669-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (830 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 02:36:01.133098 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:36:01.135006 388 webserver.cc:460] Webserver started at https://127.0.0.1:33337/ using document root <none> and password file <none>
I20260620 02:36:01.135272 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:36:01.136068 388 webserver.cc:460] Webserver started at http://127.0.0.1:45281/ using document root <none> and password file <none>
2026-06-20T02:36:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:01Z Disabled control of system clock
I20260620 02:36:01.165779 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44329
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44329
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33337/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/testchainca.cert with env {}
W20260620 02:36:01.517838 5667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:01.518183 5667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:01.518354 5667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:01.525995 5667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:01.526139 5667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:01.526177 5667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:01.526265 5667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:01.526335 5667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:36:01.526446 5667 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:36:01.526492 5667 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33337/jwks.json
I20260620 02:36:01.536968 5667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33337/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:01.538412 5667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:01.539805 5667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:01.552268 5673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:01.552354 5675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:01.552354 5672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:01.553494 5667 server_base.cc:1034] running on GCE node
I20260620 02:36:01.554404 5667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:01.556255 5667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:01.557557 5667 hybrid_clock.cc:648] HybridClock initialized: now 1781922961557489 us; error 65 us; skew 500 ppm
I20260620 02:36:01.557973 5667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:01.560618 5667 webserver.cc:460] Webserver started at http://127.0.97.62:44939/ using document root <none> and password file <none>
I20260620 02:36:01.561426 5667 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:01.561568 5667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:01.561889 5667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:01.565155 5667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "308df8757e7e46798ed9575cc2da40a3"
format_stamp: "Formatted at 2026-06-20 02:36:01 on dist-test-slave-rpg6"
server_key: "6fb06962df72713fba2091eba234dbde"
server_key_iv: "39442b3a281d3e066f194c2a86bcfda0"
server_key_version: "encryptionkey@0"
I20260620 02:36:01.566049 5667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "308df8757e7e46798ed9575cc2da40a3"
format_stamp: "Formatted at 2026-06-20 02:36:01 on dist-test-slave-rpg6"
server_key: "6fb06962df72713fba2091eba234dbde"
server_key_iv: "39442b3a281d3e066f194c2a86bcfda0"
server_key_version: "encryptionkey@0"
I20260620 02:36:01.571666 5667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:36:01.575508 5681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:01.577162 5667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:36:01.577445 5667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "308df8757e7e46798ed9575cc2da40a3"
format_stamp: "Formatted at 2026-06-20 02:36:01 on dist-test-slave-rpg6"
server_key: "6fb06962df72713fba2091eba234dbde"
server_key_iv: "39442b3a281d3e066f194c2a86bcfda0"
server_key_version: "encryptionkey@0"
I20260620 02:36:01.577672 5667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:01.598063 5667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:01.598981 5667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:01.599320 5667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:01.619917 5667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44329
I20260620 02:36:01.619925 5732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44329 every 8 connection(s)
I20260620 02:36:01.621392 5667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:01.625445 5733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:01.629452 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 5667
I20260620 02:36:01.629709 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:36:01.630507 388 external_mini_cluster.cc:1428] Setting key 459a4348f5585b15900abbc1881ef1f4
I20260620 02:36:01.638135 5733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3: Bootstrap starting.
I20260620 02:36:01.641594 5733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:01.642795 5733 log.cc:824] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:01.646023 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:01.632786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.0.97.62:44329)
0620 02:36:01.633872 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:01.633884 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:01.633931 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:36:01.636696 (+ 2765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:01.636729 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:01.637289 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:01.637795 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:01.638402 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:01.639773 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:01.641168 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:01.641982 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:01.642198 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:01.643535 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:01.643542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:01.643562 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:01.644219 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:01.644314 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:01.644343 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:01.644474 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:01.644712 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 02:36:01.645059 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":384,"threads_started":1}
I20260620 02:36:01.646286 5733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3: No bootstrap required, opened a new log
I20260620 02:36:01.650615 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:36:01.655061 5733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308df8757e7e46798ed9575cc2da40a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44329 } }
I20260620 02:36:01.655507 5733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:01.655640 5733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308df8757e7e46798ed9575cc2da40a3, State: Initialized, Role: FOLLOWER
I20260620 02:36:01.656334 5733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308df8757e7e46798ed9575cc2da40a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44329 } }
I20260620 02:36:01.656585 5733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:01.656749 5733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:01.656930 5733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:01.659314 5733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308df8757e7e46798ed9575cc2da40a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44329 } }
I20260620 02:36:01.659947 5733 leader_election.cc:302] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308df8757e7e46798ed9575cc2da40a3; no voters:
I20260620 02:36:01.660645 5733 leader_election.cc:288] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:36:01.660779 5738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:01.661712 5738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [term 1 LEADER]: Becoming Leader. State: Replica: 308df8757e7e46798ed9575cc2da40a3, State: Running, Role: LEADER
I20260620 02:36:01.662293 5738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308df8757e7e46798ed9575cc2da40a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44329 } }
I20260620 02:36:01.663129 5733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:01.665711 5740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 308df8757e7e46798ed9575cc2da40a3. Latest consensus state: current_term: 1 leader_uuid: "308df8757e7e46798ed9575cc2da40a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308df8757e7e46798ed9575cc2da40a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44329 } } }
I20260620 02:36:01.665704 5739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "308df8757e7e46798ed9575cc2da40a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308df8757e7e46798ed9575cc2da40a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44329 } } }
I20260620 02:36:01.666153 5740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:01.666201 5739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:01.668676 5744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:01.676306 5744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:01.687063 5744 catalog_manager.cc:1347] Generated new cluster ID: 22728ff44b734421986c9c794885bce5
I20260620 02:36:01.687204 5744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:01.702597 5744 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:36:01.704144 5744 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:01.719044 5744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 308df8757e7e46798ed9575cc2da40a3: Generated new TSK 0
I20260620 02:36:01.720373 5744 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 02:36:01.766716 5763 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 02:36:01.766887 5763 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:36:01.768074 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:01.746370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.0.97.62:44329)
0620 02:36:01.746568 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:01.746577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:01.747105 (+ 528us) server_negotiation.cc:407] Connection header received
0620 02:36:01.747274 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:01.747278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:01.747358 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:01.747493 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:36:01.748161 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:01.748926 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:01.751164 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:01.751669 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:01.751796 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:01.752471 (+ 675us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:36:01.767431 (+ 14960us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 02:36:01.767779 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37366: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 02:36:01.768239 5736 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37366: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 02:36:01.768481 5762 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:01.746228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44329 (local address 127.0.0.1:37366)
0620 02:36:01.746903 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:01.746924 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:01.747098 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:01.747543 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:01.747548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:01.747561 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:36:01.747902 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:01.747914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:01.749087 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:01.749093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:01.750891 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:01.750910 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:01.751852 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:01.751907 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:01.752114 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:01.752170 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:36:01.767716 (+ 15546us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 02:36:01.768075 (+ 359us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44329: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":353,"threads_started":1}
I20260620 02:36:01.769580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 5667
I20260620 02:36:01.770869 5728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:01.866029 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 5667
2026-06-20T02:36:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (790 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 02:36:01.923318 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:36:01.925468 388 webserver.cc:460] Webserver started at https://127.0.0.1:44199/ using document root <none> and password file <none>
I20260620 02:36:01.925698 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:36:01.926347 388 webserver.cc:460] Webserver started at http://127.0.0.1:45019/ using document root <none> and password file <none>
2026-06-20T02:36:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:01Z Disabled control of system clock
I20260620 02:36:01.956318 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36481
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36481
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44199/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 02:36:02.322007 5770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:02.322342 5770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:02.322654 5770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:02.330835 5770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:02.330983 5770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:02.331054 5770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:02.331167 5770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:02.331210 5770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:36:02.331390 5770 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:36:02.331446 5770 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44199/jwks.json
I20260620 02:36:02.341481 5770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44199/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:02.342947 5770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:02.344973 5770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:02.358212 5775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:02.358289 5778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:02.358224 5776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:02.359128 5770 server_base.cc:1034] running on GCE node
I20260620 02:36:02.360087 5770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:02.361776 5770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:02.365137 5770 hybrid_clock.cc:648] HybridClock initialized: now 1781922962365003 us; error 140 us; skew 500 ppm
I20260620 02:36:02.365638 5770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:02.368347 5770 webserver.cc:460] Webserver started at http://127.0.97.62:46129/ using document root <none> and password file <none>
I20260620 02:36:02.369113 5770 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:02.369210 5770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:02.369553 5770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:02.372675 5770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "a554974aa0b4434a8337ca95f796445c"
format_stamp: "Formatted at 2026-06-20 02:36:02 on dist-test-slave-rpg6"
server_key: "3e091b5292e416b926efa5d77f481238"
server_key_iv: "a4130c13d5bc43ed9a19606d39dd0dbe"
server_key_version: "encryptionkey@0"
I20260620 02:36:02.373528 5770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "a554974aa0b4434a8337ca95f796445c"
format_stamp: "Formatted at 2026-06-20 02:36:02 on dist-test-slave-rpg6"
server_key: "3e091b5292e416b926efa5d77f481238"
server_key_iv: "a4130c13d5bc43ed9a19606d39dd0dbe"
server_key_version: "encryptionkey@0"
I20260620 02:36:02.378944 5770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:36:02.382779 5784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:02.384565 5770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:36:02.384783 5770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "a554974aa0b4434a8337ca95f796445c"
format_stamp: "Formatted at 2026-06-20 02:36:02 on dist-test-slave-rpg6"
server_key: "3e091b5292e416b926efa5d77f481238"
server_key_iv: "a4130c13d5bc43ed9a19606d39dd0dbe"
server_key_version: "encryptionkey@0"
I20260620 02:36:02.385037 5770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:02.409441 5770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:02.410491 5770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:02.410809 5770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:02.432425 5770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36481
I20260620 02:36:02.432448 5835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36481 every 8 connection(s)
I20260620 02:36:02.433907 5770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:02.438043 5836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:02.439745 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 5770
I20260620 02:36:02.440093 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:36:02.440812 388 external_mini_cluster.cc:1428] Setting key 14233178b8ce3c930cc58ffd55623812
I20260620 02:36:02.452968 5836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c: Bootstrap starting.
I20260620 02:36:02.457417 5836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:02.458739 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:02.442926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.0.97.62:36481)
0620 02:36:02.444248 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:02.444269 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:02.444333 (+ 64us) server_negotiation.cc:407] Connection header received
0620 02:36:02.446891 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:02.446925 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:02.447598 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:02.448134 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:02.448770 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:02.450934 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:02.452539 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:02.453729 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:02.453927 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:02.455252 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:02.455263 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:02.455285 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:02.456262 (+ 977us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:02.456403 (+ 141us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:02.456438 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:02.456626 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:02.456879 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 02:36:02.457264 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":417,"threads_started":1}
I20260620 02:36:02.458871 5836 log.cc:824] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:02.462729 5836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c: No bootstrap required, opened a new log
I20260620 02:36:02.463770 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:36:02.471392 5836 raft_consensus.cc:348] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554974aa0b4434a8337ca95f796445c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36481 } }
I20260620 02:36:02.471932 5836 raft_consensus.cc:374] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:02.472069 5836 raft_consensus.cc:729] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a554974aa0b4434a8337ca95f796445c, State: Initialized, Role: FOLLOWER
I20260620 02:36:02.472767 5836 consensus_queue.cc:260] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554974aa0b4434a8337ca95f796445c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36481 } }
I20260620 02:36:02.473033 5836 raft_consensus.cc:388] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:02.473191 5836 raft_consensus.cc:482] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:02.473368 5836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:02.475625 5836 raft_consensus.cc:504] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554974aa0b4434a8337ca95f796445c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36481 } }
I20260620 02:36:02.476264 5836 leader_election.cc:302] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a554974aa0b4434a8337ca95f796445c; no voters:
I20260620 02:36:02.477327 5836 leader_election.cc:288] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:36:02.477504 5841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:02.478817 5841 raft_consensus.cc:686] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [term 1 LEADER]: Becoming Leader. State: Replica: a554974aa0b4434a8337ca95f796445c, State: Running, Role: LEADER
I20260620 02:36:02.479385 5841 consensus_queue.cc:237] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554974aa0b4434a8337ca95f796445c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36481 } }
I20260620 02:36:02.479892 5836 sys_catalog.cc:564] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:02.482461 5843 sys_catalog.cc:455] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a554974aa0b4434a8337ca95f796445c. Latest consensus state: current_term: 1 leader_uuid: "a554974aa0b4434a8337ca95f796445c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554974aa0b4434a8337ca95f796445c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36481 } } }
I20260620 02:36:02.482897 5843 sys_catalog.cc:458] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:02.482582 5842 sys_catalog.cc:455] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a554974aa0b4434a8337ca95f796445c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a554974aa0b4434a8337ca95f796445c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36481 } } }
I20260620 02:36:02.483024 5842 sys_catalog.cc:458] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:02.486117 5848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:02.491854 5848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:02.503099 5848 catalog_manager.cc:1347] Generated new cluster ID: 7370a3ce0250488f8f096e49e96bc7d1
I20260620 02:36:02.503248 5848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:02.515206 5848 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:36:02.516647 5848 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:02.555066 5848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a554974aa0b4434a8337ca95f796445c: Generated new TSK 0
I20260620 02:36:02.556226 5848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:36:02.595571 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:02.585742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.0.97.62:36481)
0620 02:36:02.585959 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:02.585967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:02.586650 (+ 683us) server_negotiation.cc:407] Connection header received
0620 02:36:02.587062 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:02.587066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:02.587242 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:02.587452 (+ 210us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:36:02.588024 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:02.588907 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:02.591041 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:02.591572 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:02.591681 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:02.592620 (+ 939us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:36:02.594867 (+ 2247us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44199/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 02:36:02.595241 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43302: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44199/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 02:36:02.595703 5839 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43302: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44199/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 02:36:02.595795 5865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:02.585739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36481 (local address 127.0.0.1:43302)
0620 02:36:02.586509 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:02.586532 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:02.586722 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:02.587488 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:02.587493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:02.587507 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:36:02.587861 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:02.587873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:02.589068 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:02.589074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:02.590863 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:02.590879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:02.591711 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:02.591787 (+ 76us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:02.592025 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:02.592078 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:36:02.595089 (+ 3011us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44199/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 02:36:02.595354 (+ 265us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36481: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44199/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":341,"threads_started":1}
I20260620 02:36:02.596716 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 5770
I20260620 02:36:02.597995 5831 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 02:36:02.609308 5866 webserver.cc:533] Webserver: sslize failed:
E20260620 02:36:02.609476 5866 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:36:02.694725 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 5770
2026-06-20T02:36:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (827 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T02:36:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:02Z Disabled control of system clock
I20260620 02:36:02.780390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33083
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33083
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/testchainca.cert with env {}
W20260620 02:36:03.127318 5871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:03.127657 5871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:03.127820 5871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:03.135453 5871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:03.135591 5871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:03.135629 5871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:03.135753 5871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:03.135843 5871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:36:03.135990 5871 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:36:03.136037 5871 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 02:36:03.146620 5871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33083
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:03.148165 5871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:03.149549 5871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:03.162734 5876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:03.163401 5879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:03.163615 5877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:03.164295 5871 server_base.cc:1034] running on GCE node
I20260620 02:36:03.165119 5871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:03.167119 5871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:03.168370 5871 hybrid_clock.cc:648] HybridClock initialized: now 1781922963168321 us; error 40 us; skew 500 ppm
I20260620 02:36:03.168731 5871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:03.171527 5871 webserver.cc:460] Webserver started at http://127.0.97.62:37837/ using document root <none> and password file <none>
I20260620 02:36:03.172344 5871 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:03.172442 5871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:03.172710 5871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:03.175827 5871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "60f84f367d684af8a42b8156f9749eac"
format_stamp: "Formatted at 2026-06-20 02:36:03 on dist-test-slave-rpg6"
server_key: "60727e4e35cc8017ad68d7984b978e6b"
server_key_iv: "36ab97e909b9069d871b6b5dacce12fd"
server_key_version: "encryptionkey@0"
I20260620 02:36:03.176782 5871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "60f84f367d684af8a42b8156f9749eac"
format_stamp: "Formatted at 2026-06-20 02:36:03 on dist-test-slave-rpg6"
server_key: "60727e4e35cc8017ad68d7984b978e6b"
server_key_iv: "36ab97e909b9069d871b6b5dacce12fd"
server_key_version: "encryptionkey@0"
I20260620 02:36:03.182698 5871 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:36:03.186828 5885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:03.188604 5871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:36:03.188827 5871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "60f84f367d684af8a42b8156f9749eac"
format_stamp: "Formatted at 2026-06-20 02:36:03 on dist-test-slave-rpg6"
server_key: "60727e4e35cc8017ad68d7984b978e6b"
server_key_iv: "36ab97e909b9069d871b6b5dacce12fd"
server_key_version: "encryptionkey@0"
I20260620 02:36:03.189079 5871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:03.204609 5871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:03.205523 5871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:03.205848 5871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:03.226800 5871 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33083
I20260620 02:36:03.226819 5936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33083 every 8 connection(s)
I20260620 02:36:03.228368 5871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:03.232482 5937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:03.233131 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 5871
I20260620 02:36:03.233371 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:36:03.233996 388 external_mini_cluster.cc:1428] Setting key 4a5854641fe6aa3d8742fdb261bda441
I20260620 02:36:03.248783 5937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac: Bootstrap starting.
I20260620 02:36:03.249739 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:03.236249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.0.97.62:33083)
0620 02:36:03.238026 (+ 1777us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:03.238039 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:03.238093 (+ 54us) server_negotiation.cc:407] Connection header received
0620 02:36:03.239784 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:03.239812 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:03.240533 (+ 721us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:03.241032 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:03.241676 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:03.243139 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:03.244709 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:03.245503 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:03.245791 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:03.247177 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:03.247186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:03.247203 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:03.247813 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:03.247952 (+ 139us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:03.247982 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:03.248110 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:03.248365 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 02:36:03.248702 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235,"thread_start_us":689,"threads_started":1}
I20260620 02:36:03.252732 5937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:03.253985 5937 log.cc:824] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:03.255468 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:36:03.257431 5937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac: No bootstrap required, opened a new log
I20260620 02:36:03.265060 5937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f84f367d684af8a42b8156f9749eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33083 } }
I20260620 02:36:03.265527 5937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:03.265734 5937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f84f367d684af8a42b8156f9749eac, State: Initialized, Role: FOLLOWER
I20260620 02:36:03.266464 5937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f84f367d684af8a42b8156f9749eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33083 } }
I20260620 02:36:03.266728 5937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:03.266906 5937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:03.267094 5937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:03.269526 5937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f84f367d684af8a42b8156f9749eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33083 } }
I20260620 02:36:03.270223 5937 leader_election.cc:302] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60f84f367d684af8a42b8156f9749eac; no voters:
I20260620 02:36:03.270984 5937 leader_election.cc:288] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:36:03.271481 5942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:03.272346 5942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [term 1 LEADER]: Becoming Leader. State: Replica: 60f84f367d684af8a42b8156f9749eac, State: Running, Role: LEADER
I20260620 02:36:03.272913 5942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f84f367d684af8a42b8156f9749eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33083 } }
I20260620 02:36:03.273774 5937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:03.276425 5944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60f84f367d684af8a42b8156f9749eac. Latest consensus state: current_term: 1 leader_uuid: "60f84f367d684af8a42b8156f9749eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f84f367d684af8a42b8156f9749eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33083 } } }
I20260620 02:36:03.276400 5943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60f84f367d684af8a42b8156f9749eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f84f367d684af8a42b8156f9749eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33083 } } }
I20260620 02:36:03.276983 5944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:03.277154 5943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:03.279618 5949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:03.287871 5949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:03.299783 5949 catalog_manager.cc:1347] Generated new cluster ID: adea4eef8acf4e87bfb08e6999ee970f
I20260620 02:36:03.300273 5949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:03.316018 5949 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:36:03.317932 5949 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:03.330968 5949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60f84f367d684af8a42b8156f9749eac: Generated new TSK 0
I20260620 02:36:03.332110 5949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:36:03.437263 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:03.353298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.0.97.62:33083)
0620 02:36:03.353535 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:03.353543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:03.354185 (+ 642us) server_negotiation.cc:407] Connection header received
0620 02:36:03.354414 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:03.354418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:03.354504 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:03.354654 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:36:03.355207 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:03.356118 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:03.357626 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:03.358112 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:03.358267 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:03.358804 (+ 537us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:36:03.436401 (+ 77597us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 02:36:03.436884 (+ 483us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44644: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 02:36:03.437420 5940 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44644: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 02:36:03.437459 5966 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:03.353201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33083 (local address 127.0.0.1:44644)
0620 02:36:03.354028 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:03.354049 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:03.354238 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:03.354699 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:03.354705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:03.354719 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:36:03.355049 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:03.355062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:03.356290 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:03.356296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:03.357438 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:03.357451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:03.358258 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:03.358318 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:03.358474 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:03.358512 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:36:03.436754 (+ 78242us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 02:36:03.437039 (+ 285us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33083: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":495,"threads_started":1}
I20260620 02:36:03.438558 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 5871
I20260620 02:36:03.440145 5932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:03.543625 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 5871
2026-06-20T02:36:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (828 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (433 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (451 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:36:04 dist-test-slave-rpg6 krb5kdc[5981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:36:04 dist-test-slave-rpg6 krb5kdc[5981](info): set up 2 sockets
Jun 20 02:36:04 dist-test-slave-rpg6 krb5kdc[5981](info): commencing operation
krb5kdc: starting...
W20260620 02:36:05.544734 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:36:05 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:36:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:36:05.779960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41541
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:06.136196 5994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:06.136507 5994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:06.136679 5994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:06.144078 5994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:06.144192 5994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:06.144299 5994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:06.144397 5994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:06.144479 5994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:36:06.154611 5994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:06.156008 5994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:06.157370 5994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:06.170255 5999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:06.170302 6002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:06.170207 6000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:06.173873 5994 server_base.cc:1034] running on GCE node
I20260620 02:36:06.174853 5994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:06.176549 5994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:06.177870 5994 hybrid_clock.cc:648] HybridClock initialized: now 1781922966177801 us; error 77 us; skew 500 ppm
I20260620 02:36:06.178330 5994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:06 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:06.185930 5994 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:36:06.187991 5994 webserver.cc:460] Webserver started at http://127.0.97.62:41817/ using document root <none> and password file <none>
I20260620 02:36:06.188822 5994 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:06.188977 5994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:06.189320 5994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:06.193313 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "88c8477d8eb44aa89b5e59185ef02f29"
format_stamp: "Formatted at 2026-06-20 02:36:06 on dist-test-slave-rpg6"
server_key: "43f0dc215c6ad7bb594601ae45307697"
server_key_iv: "00f499c5538a2195a0a38f4740b3423a"
server_key_version: "encryptionkey@0"
I20260620 02:36:06.194271 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "88c8477d8eb44aa89b5e59185ef02f29"
format_stamp: "Formatted at 2026-06-20 02:36:06 on dist-test-slave-rpg6"
server_key: "43f0dc215c6ad7bb594601ae45307697"
server_key_iv: "00f499c5538a2195a0a38f4740b3423a"
server_key_version: "encryptionkey@0"
I20260620 02:36:06.200256 5994 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:36:06.204457 6009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:06.206089 5994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:36:06.206282 5994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "88c8477d8eb44aa89b5e59185ef02f29"
format_stamp: "Formatted at 2026-06-20 02:36:06 on dist-test-slave-rpg6"
server_key: "43f0dc215c6ad7bb594601ae45307697"
server_key_iv: "00f499c5538a2195a0a38f4740b3423a"
server_key_version: "encryptionkey@0"
I20260620 02:36:06.206478 5994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:06.226469 5994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:06.231606 5994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:06.231912 5994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:06.252429 5994 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41541
I20260620 02:36:06.252451 6060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41541 every 8 connection(s)
I20260620 02:36:06.253914 5994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:06.257824 6061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:06.263760 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 5994
I20260620 02:36:06.264101 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:36:06.264725 388 external_mini_cluster.cc:1428] Setting key 69daf60b7640fd91736c2b846f1a5cbd
I20260620 02:36:06.269258 6061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29: Bootstrap starting.
I20260620 02:36:06.273267 6061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:06.274713 6061 log.cc:824] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:06.277976 6061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29: No bootstrap required, opened a new log
Jun 20 02:36:06 dist-test-slave-rpg6 krb5kdc[5981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:36:06.287511 6061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c8477d8eb44aa89b5e59185ef02f29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } }
I20260620 02:36:06.288086 6061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:06.288223 6061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c8477d8eb44aa89b5e59185ef02f29, State: Initialized, Role: FOLLOWER
I20260620 02:36:06.289189 6061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c8477d8eb44aa89b5e59185ef02f29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } }
I20260620 02:36:06.289585 6061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:06.289738 6061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:06.289943 6061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:06.290511 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:06.266874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.0.97.62:41541)
0620 02:36:06.267762 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:06.267775 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:06.267818 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:36:06.269735 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:06.269760 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:06.270191 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:06.270536 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:06.272485 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:06.273989 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:06.275855 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:06.276619 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:06.276824 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:06.281384 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:06.281394 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:06.281412 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:06.281459 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:06.284809 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:06.285728 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:06.285735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:06.285742 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:06.285935 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:06.286382 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:06.286386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:06.286397 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:06.287141 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:06.287382 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:06.288041 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 02:36:06.288587 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":444,"threads_started":1}
I20260620 02:36:06.293879 6061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c8477d8eb44aa89b5e59185ef02f29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } }
I20260620 02:36:06.294510 6061 leader_election.cc:302] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88c8477d8eb44aa89b5e59185ef02f29; no voters:
I20260620 02:36:06.295951 6061 leader_election.cc:288] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:36:06.296075 6066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:06.298540 6066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [term 1 LEADER]: Becoming Leader. State: Replica: 88c8477d8eb44aa89b5e59185ef02f29, State: Running, Role: LEADER
I20260620 02:36:06.298836 6061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:06.299152 6066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c8477d8eb44aa89b5e59185ef02f29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } }
I20260620 02:36:06.303665 6067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88c8477d8eb44aa89b5e59185ef02f29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c8477d8eb44aa89b5e59185ef02f29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } } }
I20260620 02:36:06.304256 6067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:06.304397 6068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88c8477d8eb44aa89b5e59185ef02f29. Latest consensus state: current_term: 1 leader_uuid: "88c8477d8eb44aa89b5e59185ef02f29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c8477d8eb44aa89b5e59185ef02f29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } } }
I20260620 02:36:06.304790 6068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:06.306465 6079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:06.311123 6079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:06.320667 6079 catalog_manager.cc:1347] Generated new cluster ID: 5a7bc37584b74ec4b6afa966e83c6c0c
I20260620 02:36:06.320816 6079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:06.335367 6079 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:36:06.336903 6079 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:06.343405 6079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88c8477d8eb44aa89b5e59185ef02f29: Generated new TSK 0
I20260620 02:36:06.344395 6079 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:36:06.397622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:06.777304 6087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:06.777612 6087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:06.777711 6087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:06.785252 6087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:06.785391 6087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:06.785506 6087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:36:06.795701 6087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:06.797183 6087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:06.798699 6087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:06.813315 6092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:06.813427 6093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:06.813355 6095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:06.814630 6087 server_base.cc:1034] running on GCE node
I20260620 02:36:06.815352 6087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:06.816568 6087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:06.818030 6087 hybrid_clock.cc:648] HybridClock initialized: now 1781922966817972 us; error 55 us; skew 500 ppm
I20260620 02:36:06.818416 6087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:06 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:06.822163 6087 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:36:06.823933 6087 webserver.cc:460] Webserver started at http://127.0.97.1:42413/ using document root <none> and password file <none>
I20260620 02:36:06.824712 6087 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:06.824868 6087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:06.825201 6087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:06.828452 6087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "2107f80810fd46b6a4e32e47fcff49f5"
format_stamp: "Formatted at 2026-06-20 02:36:06 on dist-test-slave-rpg6"
server_key: "a9006ceca405b58135d5b66c2a3b1302"
server_key_iv: "3e53438183543fd2baf2a15ffc2564d3"
server_key_version: "encryptionkey@0"
I20260620 02:36:06.829356 6087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2107f80810fd46b6a4e32e47fcff49f5"
format_stamp: "Formatted at 2026-06-20 02:36:06 on dist-test-slave-rpg6"
server_key: "a9006ceca405b58135d5b66c2a3b1302"
server_key_iv: "3e53438183543fd2baf2a15ffc2564d3"
server_key_version: "encryptionkey@0"
I20260620 02:36:06.835203 6087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:36:06.839254 6102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:06.840764 6087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:36:06.840955 6087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "2107f80810fd46b6a4e32e47fcff49f5"
format_stamp: "Formatted at 2026-06-20 02:36:06 on dist-test-slave-rpg6"
server_key: "a9006ceca405b58135d5b66c2a3b1302"
server_key_iv: "3e53438183543fd2baf2a15ffc2564d3"
server_key_version: "encryptionkey@0"
I20260620 02:36:06.841303 6087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:06.857333 6087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:06.861778 6087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:06.862092 6087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:06.863130 6087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:06.864759 6087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:06.864864 6087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:06.865016 6087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:06.865099 6087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:06.900580 6087 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41753
I20260620 02:36:06.900626 6214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41753 every 8 connection(s)
I20260620 02:36:06.902163 6087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:36:06.902637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 6087
I20260620 02:36:06.902935 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:36:06.903821 388 external_mini_cluster.cc:1428] Setting key 832a46c68e2f9fab1fff9c4600113928
Jun 20 02:36:06 dist-test-slave-rpg6 krb5kdc[5981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:36:06.926084 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:06.905221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38965 (local address 127.0.97.62:41541)
0620 02:36:06.905767 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:06.905775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:06.906967 (+ 1192us) server_negotiation.cc:407] Connection header received
0620 02:36:06.909026 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:06.909034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:06.909139 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:06.909262 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:06.911102 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:06.912169 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:06.914187 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:06.914931 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:06.915109 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:06.919334 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:06.919340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:06.919343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:06.919380 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:06.922243 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:06.923339 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:06.923344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:06.923346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:06.923481 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:06.924140 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:06.924144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:06.924147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:06.924442 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:06.924559 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:06.925641 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0620 02:36:06.925823 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":264,"threads_started":1}
I20260620 02:36:06.927124 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:06.905899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41541 (local address 127.0.97.1:38965)
0620 02:36:06.906781 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:06.906836 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:06.908728 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:06.909489 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:06.909499 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:06.910194 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:06.910887 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:06.910915 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:06.912355 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:06.912362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:06.914012 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:06.914026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:06.915210 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:06.915216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:06.915454 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:06.916304 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:06.916327 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:06.919022 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:06.922445 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:06.922457 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:06.922478 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:06.923184 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:06.923660 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:06.923668 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:06.923673 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:06.924016 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:06.924602 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:06.924616 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:06.925123 (+ 507us) client_negotiation.cc:769] Sending connection context
0620 02:36:06.925486 (+ 363us) client_negotiation.cc:240] Negotiation successful
0620 02:36:06.925958 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":368,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:36:06.928891 6215 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41541
I20260620 02:36:06.929273 6215 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:06.930047 6215 heartbeater.cc:511] Master 127.0.97.62:41541 requested a full tablet report, sending...
I20260620 02:36:06.932272 6026 ts_manager.cc:194] Registered new tserver with Master: 2107f80810fd46b6a4e32e47fcff49f5 (127.0.97.1:41753)
I20260620 02:36:06.934226 6026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:36:06.956130 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:07.303426 6221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:07.303736 6221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:07.303857 6221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:07.311712 6221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:07.311905 6221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:07.312089 6221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:36:07.322954 6221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:07.324496 6221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:07.325964 6221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:07.342397 6226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:07.342350 6229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:07.342424 6227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:07.343674 6221 server_base.cc:1034] running on GCE node
I20260620 02:36:07.344426 6221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:07.345660 6221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:07.346882 6221 hybrid_clock.cc:648] HybridClock initialized: now 1781922967346830 us; error 57 us; skew 500 ppm
I20260620 02:36:07.347177 6221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:07 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:07.350667 6221 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:36:07.352334 6221 webserver.cc:460] Webserver started at http://127.0.97.2:44371/ using document root <none> and password file <none>
I20260620 02:36:07.353009 6221 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:07.353116 6221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:07.353382 6221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:07.356539 6221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "63ddab229db24f4a822028138dfbe97c"
format_stamp: "Formatted at 2026-06-20 02:36:07 on dist-test-slave-rpg6"
server_key: "403209e5cacb1c90eeb22251599e4013"
server_key_iv: "1e6417c23431eac82e84d5cd11333ca9"
server_key_version: "encryptionkey@0"
I20260620 02:36:07.357450 6221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "63ddab229db24f4a822028138dfbe97c"
format_stamp: "Formatted at 2026-06-20 02:36:07 on dist-test-slave-rpg6"
server_key: "403209e5cacb1c90eeb22251599e4013"
server_key_iv: "1e6417c23431eac82e84d5cd11333ca9"
server_key_version: "encryptionkey@0"
I20260620 02:36:07.363090 6221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:36:07.367147 6236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:07.368907 6221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:36:07.369071 6221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "63ddab229db24f4a822028138dfbe97c"
format_stamp: "Formatted at 2026-06-20 02:36:07 on dist-test-slave-rpg6"
server_key: "403209e5cacb1c90eeb22251599e4013"
server_key_iv: "1e6417c23431eac82e84d5cd11333ca9"
server_key_version: "encryptionkey@0"
I20260620 02:36:07.369277 6221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:07.388392 6221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:07.393002 6221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:07.393297 6221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:07.394289 6221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:07.395793 6221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:07.395947 6221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 02:36:07.396112 6221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:07.396239 6221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:07.433074 6221 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39945
I20260620 02:36:07.433189 6348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39945 every 8 connection(s)
I20260620 02:36:07.434413 6221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:36:07.439101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 6221
I20260620 02:36:07.439267 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:36:07.439677 388 external_mini_cluster.cc:1428] Setting key 6a1823cfe0e136bac498087b73b46a39
Jun 20 02:36:07 dist-test-slave-rpg6 krb5kdc[5981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:36:07.457540 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:07.437154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54569 (local address 127.0.97.62:41541)
0620 02:36:07.437590 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:07.437594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:07.438687 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 02:36:07.440932 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:07.440936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:07.441002 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:07.441106 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:07.443043 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:07.443763 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:07.445638 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:07.446133 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:07.446330 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:07.450974 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:07.450980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:07.450983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:07.451019 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:07.454011 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:07.454925 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:07.454933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:07.454937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:07.455093 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:07.455589 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:07.455595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:07.455600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:07.455971 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:07.456147 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:07.456906 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 02:36:07.457146 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":213,"threads_started":1}
I20260620 02:36:07.458581 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:07.437734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41541 (local address 127.0.97.2:54569)
0620 02:36:07.438505 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:07.438551 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:07.440589 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:07.441425 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:07.441437 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:07.442196 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:07.442847 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:07.442866 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:07.443966 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:07.443971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:07.445452 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:07.445467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:07.446448 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:07.446455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:07.446708 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:07.447490 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:07.447520 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:07.450666 (+ 3146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:07.454261 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:07.454277 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:07.454293 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:07.454779 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:07.455233 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:07.455239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:07.455242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:07.455463 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:07.456104 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:07.456114 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:07.456505 (+ 391us) client_negotiation.cc:769] Sending connection context
0620 02:36:07.456829 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 02:36:07.457404 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":311,"threads_started":1}
I20260620 02:36:07.460186 6349 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41541
I20260620 02:36:07.460585 6349 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:36:07.461604 6349 heartbeater.cc:511] Master 127.0.97.62:41541 requested a full tablet report, sending...
I20260620 02:36:07.463337 6026 ts_manager.cc:194] Registered new tserver with Master: 63ddab229db24f4a822028138dfbe97c (127.0.97.2:39945)
I20260620 02:36:07.464480 6026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:36:07.492673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:07.847968 6355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:07.848582 6355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:07.849015 6355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:07.856993 6355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:07.857127 6355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:07.857316 6355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:36:07.867985 6355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:07.869372 6355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:07.870754 6355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:07.884994 6360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:07.884989 6363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:07.885037 6361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:07.886287 6355 server_base.cc:1034] running on GCE node
I20260620 02:36:07.887063 6355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:07.888257 6355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:07.889536 6355 hybrid_clock.cc:648] HybridClock initialized: now 1781922967889499 us; error 59 us; skew 500 ppm
I20260620 02:36:07.889901 6355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:07 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:07.893345 6355 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:36:07.894979 6355 webserver.cc:460] Webserver started at http://127.0.97.3:39853/ using document root <none> and password file <none>
I20260620 02:36:07.895728 6355 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:07.895926 6355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:07.896256 6355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:07.899456 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "d1ab070d249d48f082ae1aab0041e567"
format_stamp: "Formatted at 2026-06-20 02:36:07 on dist-test-slave-rpg6"
server_key: "2a06b77b71b478f7e6a006032fd91da3"
server_key_iv: "bfb9bca199f81d0f521ec26e7aa8d96e"
server_key_version: "encryptionkey@0"
I20260620 02:36:07.900461 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d1ab070d249d48f082ae1aab0041e567"
format_stamp: "Formatted at 2026-06-20 02:36:07 on dist-test-slave-rpg6"
server_key: "2a06b77b71b478f7e6a006032fd91da3"
server_key_iv: "bfb9bca199f81d0f521ec26e7aa8d96e"
server_key_version: "encryptionkey@0"
I20260620 02:36:07.906421 6355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:36:07.910460 6370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:07.912258 6355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:36:07.912467 6355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "d1ab070d249d48f082ae1aab0041e567"
format_stamp: "Formatted at 2026-06-20 02:36:07 on dist-test-slave-rpg6"
server_key: "2a06b77b71b478f7e6a006032fd91da3"
server_key_iv: "bfb9bca199f81d0f521ec26e7aa8d96e"
server_key_version: "encryptionkey@0"
I20260620 02:36:07.912707 6355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:07.929804 6355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:07.934374 6355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:07.934690 6355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:07.935750 6355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:07.937397 6215 heartbeater.cc:503] Master 127.0.97.62:41541 was elected leader, sending a full tablet report...
I20260620 02:36:07.937708 6355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:07.937820 6355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:07.938015 6355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:07.938117 6355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:07.973799 6355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36201
I20260620 02:36:07.973884 6482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36201 every 8 connection(s)
I20260620 02:36:07.975304 6355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:36:07.976714 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 6355
I20260620 02:36:07.976964 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:36:07.977424 388 external_mini_cluster.cc:1428] Setting key 002c9d515b9e52ddcc8a2c2905f33789
Jun 20 02:36:07 dist-test-slave-rpg6 krb5kdc[5981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:36:07.997804 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:07.978485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59787 (local address 127.0.97.62:41541)
0620 02:36:07.979074 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:07.979080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:07.980151 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 02:36:07.982064 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:07.982069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:07.982174 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:07.982308 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:07.983934 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:07.984749 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:07.986386 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:07.986835 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:07.986983 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:07.991013 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:07.991019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:07.991023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:07.991058 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:07.994543 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:07.995427 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:07.995431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:07.995433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:07.995556 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:07.996111 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:07.996114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:07.996117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:07.996394 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:07.996508 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:07.997303 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 02:36:07.997493 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":286,"threads_started":1}
I20260620 02:36:07.998870 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:07.979066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41541 (local address 127.0.97.3:59787)
0620 02:36:07.979990 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:07.980041 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:07.981781 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:07.982572 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:07.982581 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:07.983132 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:07.983715 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:07.983733 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:07.984934 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:07.984938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:07.986222 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:07.986235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:07.987083 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:07.987092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:07.987433 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:07.988239 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:07.988263 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:07.990751 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:07.994706 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:07.994716 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:07.994735 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:07.995266 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:07.995685 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:07.995689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:07.995691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:07.995987 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:07.996509 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:07.996516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:07.996876 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 02:36:07.997212 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 02:36:07.997645 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":318,"threads_started":1}
I20260620 02:36:08.000671 6483 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41541
I20260620 02:36:08.000989 6483 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:08.001818 6483 heartbeater.cc:511] Master 127.0.97.62:41541 requested a full tablet report, sending...
I20260620 02:36:08.003625 6026 ts_manager.cc:194] Registered new tserver with Master: d1ab070d249d48f082ae1aab0041e567 (127.0.97.3:36201)
I20260620 02:36:08.004715 6026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59787
I20260620 02:36:08.006592 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/test-admin.keytab.
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:08.038288 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 02:36:08.045444 6491 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:08.042116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.0.97.1:41753)
0620 02:36:08.043159 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:08.043167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:08.043199 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:36:08.043261 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:08.043268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:08.043505 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:08.043657 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:08.044947 (+ 1290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52476: BlockingRecv error: recv got EOF from 127.0.0.1:52476 (error 108)
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":656,"threads_started":1}
W20260620 02:36:08.045500 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:08.047710 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:08.050316 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:36:08.073815 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:08.053210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.0.97.1:41753)
0620 02:36:08.053472 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:08.053477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:08.053570 (+ 93us) server_negotiation.cc:407] Connection header received
0620 02:36:08.053857 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:08.053863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:08.054123 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:08.054290 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:08.055543 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.056770 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.058247 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.058899 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.059024 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:08.064414 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:08.064433 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:08.064459 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:08.064509 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:08.068885 (+ 4376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.070221 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.070231 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.070233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.070393 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.071121 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.071137 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.071139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.072143 (+ 1004us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:08.072469 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:08.072954 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 02:36:08.073417 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
W20260620 02:36:08.082091 6491 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:08.079592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.0.97.1:41753)
0620 02:36:08.080051 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:08.080056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:08.080521 (+ 465us) server_negotiation.cc:407] Connection header received
0620 02:36:08.080749 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:08.080759 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:08.080839 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:08.080955 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:08.081787 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52490: BlockingRecv error: recv got EOF from 127.0.0.1:52490 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
W20260620 02:36:08.082309 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:08.084409 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:08.085987 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:36:08.107527 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:08.088053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.0.97.1:41753)
0620 02:36:08.088363 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:08.088369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:08.089255 (+ 886us) server_negotiation.cc:407] Connection header received
0620 02:36:08.089450 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:08.089454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:08.089526 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:08.089642 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:08.091432 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.092343 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.094026 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.094519 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.094651 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:08.099747 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:08.099756 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:08.099761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:08.099818 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:08.103494 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.104621 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.104627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.104629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.104811 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.105666 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.105670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.105673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.106012 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:08.106321 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:08.106839 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 02:36:08.107154 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
W20260620 02:36:08.113932 6491 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:08.111782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.0.97.1:41753)
0620 02:36:08.112078 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:08.112084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:08.112366 (+ 282us) server_negotiation.cc:407] Connection header received
0620 02:36:08.112513 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:08.112517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:08.112602 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:08.112722 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:08.113580 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52514: BlockingRecv error: recv got EOF from 127.0.0.1:52514 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
W20260620 02:36:08.114040 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781922915737669-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:08.115805 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:08.117560 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:36:08.133363 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:08.119081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.0.97.1:41753)
0620 02:36:08.119339 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:08.119346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:08.119585 (+ 239us) server_negotiation.cc:407] Connection header received
0620 02:36:08.119719 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:08.119723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:08.119795 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:08.119929 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:08.121015 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.121814 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.123179 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.123594 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.123695 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:08.127881 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:08.127892 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:08.127901 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:08.127941 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:08.130701 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.131453 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.131457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.131459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.131662 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.132124 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.132127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.132130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.132403 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:08.132694 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:08.132927 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 02:36:08.133100 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 02:36:08.135389 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 6087
I20260620 02:36:08.149377 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:08.136653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.0.97.1:41753)
0620 02:36:08.136872 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:08.136879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:08.137215 (+ 336us) server_negotiation.cc:407] Connection header received
0620 02:36:08.137349 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:08.137352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:08.137426 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:08.137512 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:08.138507 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.139177 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.140533 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.140978 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.141075 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:08.143431 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:08.143439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:08.143444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:08.143497 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:08.146608 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.147325 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.147329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.147332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.147466 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.147951 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.147956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.147958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.148292 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:08.148506 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:08.148885 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 02:36:08.149055 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 02:36:08.150099 6210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:08.236959 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 6087
I20260620 02:36:08.254202 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 6221
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:36:08.273881 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:08.255133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.0.97.2:39945)
0620 02:36:08.255724 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:08.255734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:08.255967 (+ 233us) server_negotiation.cc:407] Connection header received
0620 02:36:08.256151 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:08.256160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:08.256453 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:08.256647 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:08.257598 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.259216 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.260685 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.261405 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.261512 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:08.266331 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:08.266348 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:08.266365 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:08.266411 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:08.270289 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.271269 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.271277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.271280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.271425 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.271916 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.271921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.271923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.272331 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:08.272706 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:08.273133 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 02:36:08.273434 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":232,"threads_started":1}
I20260620 02:36:08.275254 6344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:08.364953 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 6221
I20260620 02:36:08.381387 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 6355
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[5981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:36:08.398914 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:08.382630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.0.97.3:36201)
0620 02:36:08.383608 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:08.383617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:08.383669 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:36:08.383759 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:08.383766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:08.384237 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:08.384475 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:08.385570 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.386595 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.388005 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:08.388574 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:08.388722 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:08.392948 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:08.392962 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:08.392979 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:08.393017 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:08.395984 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.396721 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.396735 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.396738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.396862 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:08.397246 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:08.397251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:08.397256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:08.397603 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:08.397915 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:08.398231 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 02:36:08.398502 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":550,"threads_started":1}
I20260620 02:36:08.400409 6478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:08.485270 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 6355
I20260620 02:36:08.502426 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 5994
I20260620 02:36:08.503981 6056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:08.608996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 5994
2026-06-20T02:36:08Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4185 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[6511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[6511](info): set up 2 sockets
Jun 20 02:36:08 dist-test-slave-rpg6 krb5kdc[6511](info): commencing operation
krb5kdc: starting...
W20260620 02:36:09.744943 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:36:09 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 02:36:09.985123 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36009
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:10.336730 6524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:10.337132 6524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:10.337308 6524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:10.346608 6524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:10.346802 6524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:10.346911 6524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:10.347021 6524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:10.347119 6524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:36:10.357757 6524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:10.359292 6524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:10.360853 6524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:10.374361 6529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:10.374409 6530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:10.374321 6532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:10.375463 6524 server_base.cc:1034] running on GCE node
I20260620 02:36:10.376480 6524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:10.378127 6524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:10.379354 6524 hybrid_clock.cc:648] HybridClock initialized: now 1781922970379299 us; error 50 us; skew 500 ppm
I20260620 02:36:10.379757 6524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:10 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:10.383203 6524 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 02:36:10.385183 6524 webserver.cc:460] Webserver started at http://127.0.97.62:42225/ using document root <none> and password file <none>
I20260620 02:36:10.385944 6524 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:10.386080 6524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:10.386410 6524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:10.389657 6524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "b53dba5fad064c26a173bd4691e8a17c"
format_stamp: "Formatted at 2026-06-20 02:36:10 on dist-test-slave-rpg6"
server_key: "aa1041104b204e442999988f360449e6"
server_key_iv: "129ef80388c1d794fb3aa881603e90d6"
server_key_version: "encryptionkey@0"
I20260620 02:36:10.390479 6524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "b53dba5fad064c26a173bd4691e8a17c"
format_stamp: "Formatted at 2026-06-20 02:36:10 on dist-test-slave-rpg6"
server_key: "aa1041104b204e442999988f360449e6"
server_key_iv: "129ef80388c1d794fb3aa881603e90d6"
server_key_version: "encryptionkey@0"
I20260620 02:36:10.396310 6524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 02:36:10.400368 6539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:10.402012 6524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:36:10.402288 6524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "b53dba5fad064c26a173bd4691e8a17c"
format_stamp: "Formatted at 2026-06-20 02:36:10 on dist-test-slave-rpg6"
server_key: "aa1041104b204e442999988f360449e6"
server_key_iv: "129ef80388c1d794fb3aa881603e90d6"
server_key_version: "encryptionkey@0"
I20260620 02:36:10.402529 6524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:10.428546 6524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:10.433303 6524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:10.433642 6524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:10.452831 6524 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36009
I20260620 02:36:10.452950 6590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36009 every 8 connection(s)
I20260620 02:36:10.454488 6524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:10.458938 6591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:10.458981 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 6524
I20260620 02:36:10.459225 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:36:10.460121 388 external_mini_cluster.cc:1428] Setting key 803a6b3a610a646e03b3b2a51c2e63cc
I20260620 02:36:10.474238 6591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c: Bootstrap starting.
Jun 20 02:36:10 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:10.477851 6591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:10.479250 6591 log.cc:824] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:10.482728 6591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c: No bootstrap required, opened a new log
I20260620 02:36:10.486708 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:10.462386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.0.97.62:36009)
0620 02:36:10.463288 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:10.463303 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:10.463345 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:36:10.465776 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:10.465802 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:10.466333 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:10.466725 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:10.468729 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:10.470231 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:10.472311 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:10.473150 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:10.473399 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:10.477974 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:10.478011 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:10.478038 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:10.478103 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:10.481816 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:10.482678 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:10.482686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:10.482695 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:10.482927 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:10.483379 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:10.483387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:10.483396 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:10.484071 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:10.484428 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:10.484970 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 02:36:10.485488 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":418,"threads_started":1}
I20260620 02:36:10.492445 6591 raft_consensus.cc:348] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } }
I20260620 02:36:10.492998 6591 raft_consensus.cc:374] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:10.493110 6591 raft_consensus.cc:729] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53dba5fad064c26a173bd4691e8a17c, State: Initialized, Role: FOLLOWER
I20260620 02:36:10.493716 6591 consensus_queue.cc:260] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } }
I20260620 02:36:10.493964 6591 raft_consensus.cc:388] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:10.494117 6591 raft_consensus.cc:482] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:10.494297 6591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:10.496872 6591 raft_consensus.cc:504] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } }
I20260620 02:36:10.497473 6591 leader_election.cc:302] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b53dba5fad064c26a173bd4691e8a17c; no voters:
I20260620 02:36:10.498083 6591 leader_election.cc:288] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:36:10.498350 6596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:10.499047 6596 raft_consensus.cc:686] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 1 LEADER]: Becoming Leader. State: Replica: b53dba5fad064c26a173bd4691e8a17c, State: Running, Role: LEADER
I20260620 02:36:10.499634 6596 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } }
I20260620 02:36:10.500329 6591 sys_catalog.cc:564] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:10.502434 6597 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } }
I20260620 02:36:10.502959 6597 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:10.504391 6598 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53dba5fad064c26a173bd4691e8a17c. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } }
I20260620 02:36:10.506114 6598 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:10.509296 6601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:10.515121 6601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:10.525599 6601 catalog_manager.cc:1347] Generated new cluster ID: 909a3d673ad54206ac43ca677703694b
I20260620 02:36:10.525758 6601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:10.571978 6601 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:36:10.574077 6601 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:10.592417 6601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c: Generated new TSK 0
I20260620 02:36:10.593461 6601 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 02:36:10.660770 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36009
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:11.033968 6617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:11.034297 6617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:11.034394 6617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:11.042186 6617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:11.042326 6617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:11.042443 6617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:36:11.052702 6617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:11.054085 6617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:11.055546 6617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:11.070811 6622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:11.070777 6623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:11.070762 6625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:11.071506 6617 server_base.cc:1034] running on GCE node
I20260620 02:36:11.072315 6617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:11.073639 6617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:11.075101 6617 hybrid_clock.cc:648] HybridClock initialized: now 1781922971075036 us; error 49 us; skew 500 ppm
I20260620 02:36:11.075723 6617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:11 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:11.080070 6617 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:36:11.081921 6617 webserver.cc:460] Webserver started at http://127.0.97.1:46835/ using document root <none> and password file <none>
I20260620 02:36:11.082656 6617 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:11.082767 6617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:11.083024 6617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:11.086293 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "c62ac2bd7747422ebb0a58bcb681f6e1"
format_stamp: "Formatted at 2026-06-20 02:36:11 on dist-test-slave-rpg6"
server_key: "c18b0bc5dd7ab7780e95d62ddbbaf05a"
server_key_iv: "4521b32a07503f10d259f10709c9048c"
server_key_version: "encryptionkey@0"
I20260620 02:36:11.087334 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c62ac2bd7747422ebb0a58bcb681f6e1"
format_stamp: "Formatted at 2026-06-20 02:36:11 on dist-test-slave-rpg6"
server_key: "c18b0bc5dd7ab7780e95d62ddbbaf05a"
server_key_iv: "4521b32a07503f10d259f10709c9048c"
server_key_version: "encryptionkey@0"
I20260620 02:36:11.093248 6617 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:36:11.097234 6632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:11.098958 6617 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:36:11.099174 6617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "c62ac2bd7747422ebb0a58bcb681f6e1"
format_stamp: "Formatted at 2026-06-20 02:36:11 on dist-test-slave-rpg6"
server_key: "c18b0bc5dd7ab7780e95d62ddbbaf05a"
server_key_iv: "4521b32a07503f10d259f10709c9048c"
server_key_version: "encryptionkey@0"
I20260620 02:36:11.099424 6617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:11.138047 6617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:11.142863 6617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:11.143222 6617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:11.144363 6617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:11.145805 6617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:11.145925 6617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:11.146083 6617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:11.146158 6617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:11.183760 6617 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42313
I20260620 02:36:11.183820 6744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42313 every 8 connection(s)
I20260620 02:36:11.185205 6617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:36:11.188360 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 6617
I20260620 02:36:11.188616 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:36:11.189333 388 external_mini_cluster.cc:1428] Setting key eba121eff7509d5224bffc07f190da70
Jun 20 02:36:11 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:36:11.213332 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:11.188439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52063 (local address 127.0.97.62:36009)
0620 02:36:11.188952 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:11.188959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:11.190300 (+ 1341us) server_negotiation.cc:407] Connection header received
0620 02:36:11.192637 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:11.192642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:11.192733 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:11.192871 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:11.194535 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:11.195284 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:11.197111 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:11.197547 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:11.197704 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:11.205405 (+ 7701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:11.205413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:11.205416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:11.205458 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:11.209182 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:11.210354 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:11.210368 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:11.210372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:11.210509 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:11.211082 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:11.211096 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:11.211099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:11.211410 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:11.211588 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:11.212609 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0620 02:36:11.212893 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":242,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260620 02:36:11.214481 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:11.189172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.1:52063)
0620 02:36:11.190108 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:11.190162 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:11.192328 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:11.193131 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:11.193141 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:11.193714 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:11.194336 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:11.194359 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:11.195471 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:11.195475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:11.196933 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:11.196947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:11.197849 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:11.197859 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:11.198093 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:11.202137 (+ 4044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:11.202175 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:11.205085 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:11.209429 (+ 4344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:11.209441 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:11.209472 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:11.210196 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:11.210644 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:11.210650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:11.210654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:11.210944 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:11.211571 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:11.211581 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:11.212137 (+ 556us) client_negotiation.cc:769] Sending connection context
0620 02:36:11.212472 (+ 335us) client_negotiation.cc:240] Negotiation successful
0620 02:36:11.213151 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":398,"threads_started":1}
I20260620 02:36:11.216583 6745 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36009
I20260620 02:36:11.216988 6745 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:11.217906 6745 heartbeater.cc:511] Master 127.0.97.62:36009 requested a full tablet report, sending...
I20260620 02:36:11.220551 6556 ts_manager.cc:194] Registered new tserver with Master: c62ac2bd7747422ebb0a58bcb681f6e1 (127.0.97.1:42313)
I20260620 02:36:11.222102 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 02:36:11.245937 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36009
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:11.598026 6751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:11.598345 6751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:11.598518 6751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:11.606367 6751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:11.606503 6751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:11.606674 6751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:36:11.617374 6751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:11.618752 6751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:11.620127 6751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:11.634177 6756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:11.634265 6757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:11.634423 6759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:11.636487 6751 server_base.cc:1034] running on GCE node
I20260620 02:36:11.637236 6751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:11.638335 6751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:11.639621 6751 hybrid_clock.cc:648] HybridClock initialized: now 1781922971639573 us; error 40 us; skew 500 ppm
I20260620 02:36:11.640137 6751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:11 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:11.643617 6751 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:36:11.645303 6751 webserver.cc:460] Webserver started at http://127.0.97.2:39777/ using document root <none> and password file <none>
I20260620 02:36:11.646054 6751 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:11.646214 6751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:11.646528 6751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:11.649693 6751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "94cd1db181e54aeca840315161cb9e67"
format_stamp: "Formatted at 2026-06-20 02:36:11 on dist-test-slave-rpg6"
server_key: "c32df84751588665fa5d889d503d3c4b"
server_key_iv: "1277c7b58c54e2058819fdfc1bcaf795"
server_key_version: "encryptionkey@0"
I20260620 02:36:11.650524 6751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "94cd1db181e54aeca840315161cb9e67"
format_stamp: "Formatted at 2026-06-20 02:36:11 on dist-test-slave-rpg6"
server_key: "c32df84751588665fa5d889d503d3c4b"
server_key_iv: "1277c7b58c54e2058819fdfc1bcaf795"
server_key_version: "encryptionkey@0"
I20260620 02:36:11.656085 6751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:36:11.659987 6766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:11.661433 6751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:36:11.661643 6751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "94cd1db181e54aeca840315161cb9e67"
format_stamp: "Formatted at 2026-06-20 02:36:11 on dist-test-slave-rpg6"
server_key: "c32df84751588665fa5d889d503d3c4b"
server_key_iv: "1277c7b58c54e2058819fdfc1bcaf795"
server_key_version: "encryptionkey@0"
I20260620 02:36:11.661877 6751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:11.693889 6751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:11.699132 6751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:11.699632 6751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:11.701179 6751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:11.702749 6751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:11.702875 6751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:11.703030 6751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:11.703079 6751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:11.741206 6751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41019
I20260620 02:36:11.741308 6878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41019 every 8 connection(s)
I20260620 02:36:11.742604 6751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:36:11.750241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 6751
I20260620 02:36:11.750541 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:36:11.751015 388 external_mini_cluster.cc:1428] Setting key e907d26d7b72ac4fd077a2b77a171661
Jun 20 02:36:11 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:11.764508 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:11.745381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44803 (local address 127.0.97.62:36009)
0620 02:36:11.745919 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:11.745923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:11.746768 (+ 845us) server_negotiation.cc:407] Connection header received
0620 02:36:11.748817 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:11.748823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:11.748917 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:11.749047 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:11.750692 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:11.751508 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:11.753182 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:11.753638 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:11.753776 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:11.758113 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:11.758119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:11.758122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:11.758160 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:11.761241 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:11.762108 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:11.762114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:11.762117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:11.762254 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:11.762702 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:11.762718 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:11.762721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:11.762990 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:11.763145 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:11.763902 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 02:36:11.764134 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":329,"threads_started":1}
I20260620 02:36:11.765558 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:11.745847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.2:44803)
0620 02:36:11.746571 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:11.746618 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:11.748509 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:11.749293 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:11.749301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:11.749842 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:11.750502 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:11.750527 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:11.751661 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:11.751666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:11.753023 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:11.753036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:11.753882 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:11.753888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:11.754130 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:11.754930 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:11.754954 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:11.757808 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:11.761448 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:11.761459 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:11.761487 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:11.761949 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:11.762373 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:11.762378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:11.762382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:11.762580 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:11.763122 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:11.763131 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:11.763503 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 02:36:11.763775 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 02:36:11.764373 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":288,"threads_started":1}
I20260620 02:36:11.767269 6879 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36009
I20260620 02:36:11.767608 6879 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:11.768477 6879 heartbeater.cc:511] Master 127.0.97.62:36009 requested a full tablet report, sending...
I20260620 02:36:11.770232 6556 ts_manager.cc:194] Registered new tserver with Master: 94cd1db181e54aeca840315161cb9e67 (127.0.97.2:41019)
I20260620 02:36:11.771541 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44803
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:36:11.805570 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36009
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:12.162592 6885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:12.162909 6885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:12.163085 6885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:12.171052 6885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:12.171193 6885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:12.171391 6885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:36:12.182009 6885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:12.183511 6885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:12.184995 6885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:12.201392 6891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:12.201598 6893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:12.201701 6890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:12.201905 6885 server_base.cc:1034] running on GCE node
I20260620 02:36:12.202901 6885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:12.204228 6885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:12.205524 6885 hybrid_clock.cc:648] HybridClock initialized: now 1781922972205466 us; error 48 us; skew 500 ppm
I20260620 02:36:12.205971 6885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:12 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:12.210242 6885 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:36:12.212097 6885 webserver.cc:460] Webserver started at http://127.0.97.3:37691/ using document root <none> and password file <none>
I20260620 02:36:12.212889 6885 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:12.213057 6885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:12.213398 6885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:12.216815 6885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "056f42312e944df685297ea3645a0ede"
format_stamp: "Formatted at 2026-06-20 02:36:12 on dist-test-slave-rpg6"
server_key: "4e55a485b99035184ff0ccc6128cee1e"
server_key_iv: "8b5040725a29ed891d040d8f6e95e433"
server_key_version: "encryptionkey@0"
I20260620 02:36:12.217835 6885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "056f42312e944df685297ea3645a0ede"
format_stamp: "Formatted at 2026-06-20 02:36:12 on dist-test-slave-rpg6"
server_key: "4e55a485b99035184ff0ccc6128cee1e"
server_key_iv: "8b5040725a29ed891d040d8f6e95e433"
server_key_version: "encryptionkey@0"
I20260620 02:36:12.223712 6885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:36:12.225605 6745 heartbeater.cc:503] Master 127.0.97.62:36009 was elected leader, sending a full tablet report...
I20260620 02:36:12.228807 6900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:12.230496 6885 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260620 02:36:12.230846 6885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "056f42312e944df685297ea3645a0ede"
format_stamp: "Formatted at 2026-06-20 02:36:12 on dist-test-slave-rpg6"
server_key: "4e55a485b99035184ff0ccc6128cee1e"
server_key_iv: "8b5040725a29ed891d040d8f6e95e433"
server_key_version: "encryptionkey@0"
I20260620 02:36:12.231397 6885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:12.249969 6885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:12.255726 6885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:12.256124 6885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:12.257242 6885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:12.258843 6885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:12.258955 6885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:12.259115 6885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:12.259195 6885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:12.303493 6885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34605
I20260620 02:36:12.303530 7012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34605 every 8 connection(s)
I20260620 02:36:12.305085 6885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:36:12.310328 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 6885
I20260620 02:36:12.310542 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:36:12.311026 388 external_mini_cluster.cc:1428] Setting key 647f8eaf93ba1f3265dae6ec38a6c434
Jun 20 02:36:12 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:12.327883 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.308642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54445 (local address 127.0.97.62:36009)
0620 02:36:12.309245 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:12.309249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:12.310360 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 02:36:12.312581 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:12.312587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:12.312696 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:12.312822 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:12.314458 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.315250 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.316923 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.317332 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.317472 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:12.321389 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:12.321394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:12.321398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:12.321433 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:12.324446 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.325335 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.325340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.325343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.325477 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.325950 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.325954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.325957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.326231 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:12.326402 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:12.327210 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 02:36:12.327461 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":298,"threads_started":1}
I20260620 02:36:12.328938 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.309242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.3:54445)
0620 02:36:12.310120 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:12.310225 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:12.312287 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:12.313087 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:12.313097 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:12.313651 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:12.314281 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:12.314300 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.315426 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.315430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:12.316770 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:12.316783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.317576 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.317583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:12.317847 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:12.318581 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:12.318604 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:12.321104 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:12.324649 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:12.324660 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:12.324694 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:12.325169 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:12.325596 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:12.325602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:12.325606 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:12.325816 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:12.326386 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:12.326395 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:12.326808 (+ 413us) client_negotiation.cc:769] Sending connection context
0620 02:36:12.327098 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 02:36:12.327661 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":298,"threads_started":1}
I20260620 02:36:12.330868 7013 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36009
I20260620 02:36:12.331328 7013 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:12.332237 7013 heartbeater.cc:511] Master 127.0.97.62:36009 requested a full tablet report, sending...
I20260620 02:36:12.334139 6556 ts_manager.cc:194] Registered new tserver with Master: 056f42312e944df685297ea3645a0ede (127.0.97.3:34605)
I20260620 02:36:12.335194 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54445
I20260620 02:36:12.341050 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260620 02:36:12.397380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44071
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36009,127.0.97.61:44071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:12.749686 7019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:12.750046 7019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:12.750216 7019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:12.758158 7019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:12.758356 7019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:12.758474 7019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:12.758571 7019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:12.758775 7019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:36:12.769910 7019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36009,127.0.97.61:44071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
W20260620 02:36:12.770275 7019 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:36009,127.0.97.61:44071'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 02:36:12.771363 7019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:12.772789 7019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 02:36:12.774556 6879 heartbeater.cc:503] Master 127.0.97.62:36009 was elected leader, sending a full tablet report...
W20260620 02:36:12.786976 7024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:12.786921 7027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:12.786924 7025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:12.787809 7019 server_base.cc:1034] running on GCE node
I20260620 02:36:12.788800 7019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:12.790438 7019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:12.791728 7019 hybrid_clock.cc:648] HybridClock initialized: now 1781922972791647 us; error 61 us; skew 500 ppm
I20260620 02:36:12.792179 7019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:12 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:12.795917 7019 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 02:36:12.797636 7019 webserver.cc:460] Webserver started at http://127.0.97.61:34851/ using document root <none> and password file <none>
I20260620 02:36:12.798415 7019 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:12.798555 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:12.798873 7019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:12.802309 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data/instance:
uuid: "f48ad3e294f44c9ba44e8baa08401195"
format_stamp: "Formatted at 2026-06-20 02:36:12 on dist-test-slave-rpg6"
server_key: "bae2a7952094526ea0707c80326c3829"
server_key_iv: "1866436a272ec4f85dd994ca7ac85f5e"
server_key_version: "encryptionkey@0"
I20260620 02:36:12.803757 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/wal/instance:
uuid: "f48ad3e294f44c9ba44e8baa08401195"
format_stamp: "Formatted at 2026-06-20 02:36:12 on dist-test-slave-rpg6"
server_key: "bae2a7952094526ea0707c80326c3829"
server_key_iv: "1866436a272ec4f85dd994ca7ac85f5e"
server_key_version: "encryptionkey@0"
I20260620 02:36:12.810700 7019 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:36:12.814738 7034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:12.816606 7019 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 02:36:12.816836 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/wal
uuid: "f48ad3e294f44c9ba44e8baa08401195"
format_stamp: "Formatted at 2026-06-20 02:36:12 on dist-test-slave-rpg6"
server_key: "bae2a7952094526ea0707c80326c3829"
server_key_iv: "1866436a272ec4f85dd994ca7ac85f5e"
server_key_version: "encryptionkey@0"
I20260620 02:36:12.817101 7019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:12.837527 7019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:12.842394 7019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:12.842716 7019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:12.863027 7019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44071
I20260620 02:36:12.863047 7085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44071 every 8 connection(s)
I20260620 02:36:12.864562 7019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:36:12.868872 7086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:12.871373 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 7019
I20260620 02:36:12.877182 7086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:36:12 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 02:36:12 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:12.896011 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.879859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49140 (local address 127.0.97.62:36009)
0620 02:36:12.880466 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:12.880474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:12.880918 (+ 444us) server_negotiation.cc:407] Connection header received
0620 02:36:12.881104 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:12.881109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:12.881171 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:12.881302 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:12.882513 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.883269 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.884925 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.885355 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.885497 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:12.889284 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:12.889290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:12.889293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:12.889333 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:12.893016 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.893797 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.893801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.893804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.893924 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.894336 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.894345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.894348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.894683 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:12.894841 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:12.895385 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 02:36:12.895624 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":263,"threads_started":1}
I20260620 02:36:12.897202 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.873722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.0.97.61:44071)
0620 02:36:12.874677 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:12.874692 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:12.874737 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:36:12.876712 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:12.876738 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:12.877187 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:12.877672 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:12.879191 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.881080 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.882363 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.883066 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.883281 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:12.889051 (+ 5770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:12.889062 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:12.889077 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:12.889111 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:12.892815 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.893773 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.893784 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.893788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.893966 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.894395 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.894402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.894406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.894996 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:12.895326 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:12.895693 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 02:36:12.896071 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":715,"thread_start_us":457,"threads_started":1}
I20260620 02:36:12.897238 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.879888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49140)
0620 02:36:12.880767 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:12.880811 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:12.880971 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:12.881363 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:12.881371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:12.882080 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:12.882349 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:12.882359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.883413 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.883418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:12.884776 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:12.884789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.885613 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.885623 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:12.885792 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:12.886437 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:12.886462 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:12.889054 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:12.893189 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:12.893203 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:12.893216 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:12.893650 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:12.894037 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:12.894042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:12.894046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:12.894217 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:12.894818 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:12.894827 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:12.895030 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 02:36:12.895295 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 02:36:12.895894 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":459,"threads_started":1}
I20260620 02:36:12.899538 7086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:36:12 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922972, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:36:12.914379 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.900965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44071 (local address 127.0.0.1:35178)
0620 02:36:12.901211 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:12.901235 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:12.901351 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:12.901599 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:12.901621 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:12.901964 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:12.902098 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:12.902108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.903307 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.903314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:12.904723 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:12.904746 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.905629 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.905634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:12.905811 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:12.906415 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:12.906434 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:12.908906 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:12.912014 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:12.912018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:12.912021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:12.912473 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:12.912945 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:12.912950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:12.912953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:12.913082 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:12.913599 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:12.913602 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:12.913745 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 02:36:12.913868 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 02:36:12.914113 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 02:36:12.914589 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.900855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.0.97.61:44071)
0620 02:36:12.901047 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:12.901053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:12.901296 (+ 243us) server_negotiation.cc:407] Connection header received
0620 02:36:12.901441 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:12.901445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:12.901500 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:12.901581 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:12.902293 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.903028 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.904949 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.905423 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.905548 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:12.909117 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:12.909122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:12.909125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:12.909193 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:12.911793 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.912646 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.912650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.912652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.912813 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.913190 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.913195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.913198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.913488 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:12.913598 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:12.913981 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 02:36:12.914285 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":61}
I20260620 02:36:12.919646 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: Bootstrap starting.
I20260620 02:36:12.923208 7086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:12.924330 7086 log.cc:824] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:12.927436 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: No bootstrap required, opened a new log
I20260620 02:36:12.935743 7086 raft_consensus.cc:348] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } }
I20260620 02:36:12.936244 7086 raft_consensus.cc:374] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:12.936329 7086 raft_consensus.cc:729] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48ad3e294f44c9ba44e8baa08401195, State: Initialized, Role: FOLLOWER
I20260620 02:36:12.937093 7086 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } }
I20260620 02:36:12.938053 7093 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } } }
I20260620 02:36:12.938498 7093 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:36:12.939169 7086 sys_catalog.cc:564] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:36:12.949417 7104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:36:12.949582 7104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:36:12.951426 7019 master_runner.cc:428] Detected that this master f48ad3e294f44c9ba44e8baa08401195 is joining an existing cluster
I20260620 02:36:12.951560 7019 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:44071
I20260620 02:36:12.980582 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.960411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.0.97.61:44071)
0620 02:36:12.960686 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:12.960692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:12.962799 (+ 2107us) server_negotiation.cc:407] Connection header received
0620 02:36:12.962996 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:12.963002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:12.963073 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:12.963187 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:12.965243 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.966874 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.969077 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.969780 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.969884 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:12.973444 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:12.973451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:12.973455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:12.973489 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:12.977349 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.978263 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.978269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.978272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.978423 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.978833 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.978838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.978841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.979221 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:12.979376 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:12.979954 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 02:36:12.980215 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 02:36:12.982012 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.960697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44071 (local address 127.0.0.1:35194)
0620 02:36:12.962662 (+ 1965us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:12.962686 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:12.962821 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:12.963341 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:12.963348 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:12.964234 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:12.964850 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:12.964865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.967171 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.967177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:12.968911 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:12.968925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.970221 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.970228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:12.970431 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:12.971645 (+ 1214us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:12.971676 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:12.973233 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:12.977544 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:12.977549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:12.977553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:12.978124 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:12.978562 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:12.978567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:12.978571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:12.978721 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:12.979450 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:12.979455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:12.979624 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 02:36:12.981192 (+ 1568us) client_negotiation.cc:240] Negotiation successful
0620 02:36:12.981523 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"mutex_wait_us":60,"spinlock_wait_cycles":73216,"thread_start_us":1413,"threads_started":1}
I20260620 02:36:12.982378 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.959808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.0.97.62:36009)
0620 02:36:12.960079 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:12.960085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:12.962100 (+ 2015us) server_negotiation.cc:407] Connection header received
0620 02:36:12.962453 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:12.962672 (+ 219us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:12.962845 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:12.962943 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:12.964869 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.966166 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.968851 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.969666 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.969790 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:12.973673 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:12.973679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:12.973684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:12.973726 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:12.977812 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.978587 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.978595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.978602 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.978764 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:12.980450 (+ 1686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:12.980457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:12.980461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:12.980866 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:12.981019 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:12.981690 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 02:36:12.981964 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 02:36:12.983673 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.960405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49146)
0620 02:36:12.961932 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:12.961957 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:12.962270 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:12.963202 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:12.963206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:12.964089 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:12.964663 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:12.964679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.966402 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.966409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:12.968651 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:12.968669 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:12.970128 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:12.970137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:12.970423 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:12.971544 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:12.971581 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:12.973420 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:12.978007 (+ 4587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:12.978013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:12.978017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:12.978436 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:12.978925 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:12.978933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:12.978940 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:12.979132 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:12.981107 (+ 1975us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:12.981112 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:12.981363 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 02:36:12.982769 (+ 1406us) client_negotiation.cc:240] Negotiation successful
0620 02:36:12.983131 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1368,"mutex_wait_us":54,"thread_start_us":1050,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 20 02:36:13 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:36:13.016992 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.990566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.0.97.61:44071)
0620 02:36:12.990794 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:12.990800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:12.996713 (+ 5913us) server_negotiation.cc:407] Connection header received
0620 02:36:12.997033 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:12.997042 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:12.997175 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:12.997296 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:12.998834 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.000167 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.002548 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.003494 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.003654 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.009446 (+ 5792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.009454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.009460 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.009508 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.013098 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.014182 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.014191 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.014198 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.014393 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.014979 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.014987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.014992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.015555 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.015721 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.016334 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.016590 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:36:13.017079 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:12.990568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44071 (local address 127.0.0.1:35210)
0620 02:36:12.996493 (+ 5925us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:12.996541 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:12.996790 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:12.997373 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:12.997382 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:12.998324 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:12.998629 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:12.998648 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.000334 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.000340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.002320 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.002340 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.003948 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.003957 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.004333 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.005383 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:13.005411 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:13.009109 (+ 3698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:13.013290 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.013311 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.013321 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.013981 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.014567 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.014574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.014607 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.014835 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.015703 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:13.015713 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:13.016026 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 02:36:13.016242 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 02:36:13.016667 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5631,"thread_start_us":5483,"threads_started":1}
I20260620 02:36:13.019896 6556 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:44071: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 44071
}
attrs {
promote: true
}
}
dest_uuid: "b53dba5fad064c26a173bd4691e8a17c"
cas_config_opid_index: -1
I20260620 02:36:13.021119 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260620 02:36:13.025146 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42509
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36009,127.0.97.61:44071,127.0.97.60:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:36:13.034801 7117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f48ad3e294f44c9ba44e8baa08401195 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } } }
I20260620 02:36:13.036428 7061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:13.039558 7117 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:44071
I20260620 02:36:13.040448 7019 master.cc:537] Master@127.0.97.61:44071 shutting down...
I20260620 02:36:13.041661 7116 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } } }
I20260620 02:36:13.042070 7116 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:13.040342 7061 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Refusing update from remote peer b53dba5fad064c26a173bd4691e8a17c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:36:13.050436 7117 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:36:13.073045 6543 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 02:36:13.074018 7019 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:13.074460 7019 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:13.074654 7019 tablet_replica.cc:330] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: stopping tablet replica
W20260620 02:36:13.075232 6543 consensus_peers.cc:487] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c -> Peer f48ad3e294f44c9ba44e8baa08401195 (127.0.97.61:44071): Couldn't send request to peer f48ad3e294f44c9ba44e8baa08401195. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:36:13.081956 7019 master.cc:552] Master@127.0.97.61:44071 shutdown complete.
I20260620 02:36:13.082064 7019 master_runner.cc:305] Clearing existing system tablet
I20260620 02:36:13.082594 7019 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:13.082815 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:13.086216 7019 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 02:36:13.088444 7122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:13.088912 7019 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:13.089202 7019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 02:36:13.089329 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/wal
uuid: "f48ad3e294f44c9ba44e8baa08401195"
format_stamp: "Formatted at 2026-06-20 02:36:12 on dist-test-slave-rpg6"
server_key: "bae2a7952094526ea0707c80326c3829"
server_key_iv: "1866436a272ec4f85dd994ca7ac85f5e"
server_key_version: "encryptionkey@0"
I20260620 02:36:13.095207 7019 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:13.102692 7019 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:36:13.102905 7019 log.cc:1196] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 02:36:13.103451 7019 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: Deleting consensus metadata
I20260620 02:36:13.103722 7019 master_runner.cc:315] Copying system tablet from 127.0.97.62:36009
I20260620 02:36:13.109529 7019 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:36009
I20260620 02:36:13.136142 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.110407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49148)
0620 02:36:13.120402 (+ 9995us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.120425 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:13.120589 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:13.121042 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:13.121047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:13.121573 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:13.121889 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.121906 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.123463 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.123469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.125145 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.125167 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.126307 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.126313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.126505 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.127364 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:13.127388 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:13.128575 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:13.132920 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.132928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.132933 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.133519 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.134089 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.134098 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.134102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.134266 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.134932 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:13.134937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:13.135137 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 02:36:13.135302 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 02:36:13.135639 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9823,"thread_start_us":9605,"threads_started":1}
I20260620 02:36:13.136193 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.110443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49148 (local address 127.0.97.62:36009)
0620 02:36:13.111105 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.111115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.120570 (+ 9455us) server_negotiation.cc:407] Connection header received
0620 02:36:13.120721 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.120732 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.120842 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.120973 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:13.122096 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.123231 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.125352 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.126098 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.126274 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.128826 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.128834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.128840 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.128883 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.132710 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.133788 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.133794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.133799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.133957 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.134427 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.134432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.134436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.134800 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.134952 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.135557 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.135785 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287}
I20260620 02:36:13.137583 6576 tablet_copy_service.cc:140] P b53dba5fad064c26a173bd4691e8a17c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f48ad3e294f44c9ba44e8baa08401195 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:49148)
I20260620 02:36:13.137880 6576 tablet_copy_service.cc:161] P b53dba5fad064c26a173bd4691e8a17c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f48ad3e294f44c9ba44e8baa08401195 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:49148: session id = f48ad3e294f44c9ba44e8baa08401195-00000000000000000000000000000000
I20260620 02:36:13.142693 6576 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:36:13.145334 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:36:13.152798 7019 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: tablet copy: Starting download of 0 data blocks...
I20260620 02:36:13.153148 7019 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: tablet copy: Starting download of 1 WAL segments...
I20260620 02:36:13.156278 7019 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:36:13.161486 6576 tablet_copy_service.cc:342] P b53dba5fad064c26a173bd4691e8a17c: Request end of tablet copy session f48ad3e294f44c9ba44e8baa08401195-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:49148
I20260620 02:36:13.161756 6576 tablet_copy_service.cc:433] P b53dba5fad064c26a173bd4691e8a17c: ending tablet copy session f48ad3e294f44c9ba44e8baa08401195-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f48ad3e294f44c9ba44e8baa08401195
I20260620 02:36:13.176301 7019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:13.179504 7129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:13.180042 7130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:13.187124 7132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:13.191731 7019 server_base.cc:1034] running on GCE node
I20260620 02:36:13.192278 7019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:13.192961 7019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:13.194284 7019 hybrid_clock.cc:648] HybridClock initialized: now 1781922973194218 us; error 38 us; skew 500 ppm
I20260620 02:36:13.194522 7019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:13 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:13.197400 7019 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 02:36:13.199021 7019 webserver.cc:460] Webserver started at http://127.0.97.61:35273/ using document root <none> and password file <none>
I20260620 02:36:13.199404 7019 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:13.199486 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:13.202879 7019 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 02:36:13.205286 7139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:13.205888 7019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 02:36:13.206039 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/wal
uuid: "f48ad3e294f44c9ba44e8baa08401195"
format_stamp: "Formatted at 2026-06-20 02:36:12 on dist-test-slave-rpg6"
server_key: "bae2a7952094526ea0707c80326c3829"
server_key_iv: "1866436a272ec4f85dd994ca7ac85f5e"
server_key_version: "encryptionkey@0"
I20260620 02:36:13.206283 7019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:13.222118 7019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:13.226545 7019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:13.252620 7019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44071
I20260620 02:36:13.252704 7190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44071 every 8 connection(s)
I20260620 02:36:13.253757 7019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:36:13.260994 7191 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:36:13.265282 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: Bootstrap starting.
I20260620 02:36:13.286478 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:36:13.287160 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: Bootstrap complete.
I20260620 02:36:13.288626 7191 raft_consensus.cc:348] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } }
I20260620 02:36:13.289018 7191 raft_consensus.cc:729] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f48ad3e294f44c9ba44e8baa08401195, State: Initialized, Role: LEARNER
I20260620 02:36:13.289333 7191 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } }
I20260620 02:36:13.290373 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } } }
I20260620 02:36:13.290694 7191 sys_catalog.cc:564] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:13.290879 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:36:13.301657 7205 catalog_manager.cc:1259] Loaded cluster ID: 909a3d673ad54206ac43ca677703694b
I20260620 02:36:13.301786 7205 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: loading cluster ID for follower catalog manager: success
I20260620 02:36:13.305511 7205 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: acquiring CA information for follower catalog manager: success
I20260620 02:36:13.309096 7205 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:36:13.338855 7013 heartbeater.cc:503] Master 127.0.97.62:36009 was elected leader, sending a full tablet report...
W20260620 02:36:13.476011 7118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:13.476364 7118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:13.476459 7118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:13.485286 7118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:13.485423 7118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:13.485471 7118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:13.485527 7118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:13.485565 7118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:36:13.497339 7118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36009,127.0.97.61:44071,127.0.97.60:42509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:13.498795 7118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:13.500211 7118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:13.514473 7210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:13.514503 7211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:13.514443 7213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:13.515376 7118 server_base.cc:1034] running on GCE node
I20260620 02:36:13.516676 7118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:13.518368 7118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:13.519605 7118 hybrid_clock.cc:648] HybridClock initialized: now 1781922973519571 us; error 42 us; skew 500 ppm
I20260620 02:36:13.520171 7118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:13 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:13.524241 7118 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 02:36:13.526156 7118 webserver.cc:460] Webserver started at http://127.0.97.60:41655/ using document root <none> and password file <none>
I20260620 02:36:13.526970 7118 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:13.527141 7118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:13.527463 7118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:13.531852 7118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data/instance:
uuid: "809c672b0cdc4956ada9756e748673bc"
format_stamp: "Formatted at 2026-06-20 02:36:13 on dist-test-slave-rpg6"
server_key: "77bf399fcced37806980944310991b9b"
server_key_iv: "2e3b7f18b1598c7b926e13164dc23425"
server_key_version: "encryptionkey@0"
I20260620 02:36:13.532951 7118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/wal/instance:
uuid: "809c672b0cdc4956ada9756e748673bc"
format_stamp: "Formatted at 2026-06-20 02:36:13 on dist-test-slave-rpg6"
server_key: "77bf399fcced37806980944310991b9b"
server_key_iv: "2e3b7f18b1598c7b926e13164dc23425"
server_key_version: "encryptionkey@0"
I20260620 02:36:13.537420 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.531013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.0.97.61:44071)
0620 02:36:13.531642 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.531648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.531669 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:36:13.531752 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.531757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.531887 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.532027 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:13.532506 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.533632 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.535247 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.536282 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.536444 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.536690 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.536935 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.537068 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":319,"threads_started":1}
I20260620 02:36:13.537420 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.530710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44071 (local address 127.0.0.1:35224)
0620 02:36:13.531285 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.531314 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:13.531476 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:13.532082 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:13.532086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:13.532110 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:13.532327 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.532336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.533799 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.533805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.535083 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.535097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.536490 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.536496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.536737 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.536791 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 02:36:13.536908 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 02:36:13.537060 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":248,"threads_started":1}
I20260620 02:36:13.538888 7166 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 02:36:13.539440 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53dba5fad064c26a173bd4691e8a17c. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } } }
I20260620 02:36:13.539695 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:36:13.539763 7118 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:36:13.543579 7222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:13.545146 7118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:36:13.545380 7118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/wal
uuid: "809c672b0cdc4956ada9756e748673bc"
format_stamp: "Formatted at 2026-06-20 02:36:13 on dist-test-slave-rpg6"
server_key: "77bf399fcced37806980944310991b9b"
server_key_iv: "2e3b7f18b1598c7b926e13164dc23425"
server_key_version: "encryptionkey@0"
I20260620 02:36:13.545594 7118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:13.570597 7118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:13.575555 7118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:13.575933 7118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:13.599107 7118 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:42509
I20260620 02:36:13.599128 7273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:42509 every 8 connection(s)
I20260620 02:36:13.600764 7118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:36:13.603422 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 7118
I20260620 02:36:13.605862 7274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:13.615440 7274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:36:13 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 02:36:13 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:13.629802 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.606127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.0.97.60:42509)
0620 02:36:13.607189 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.607206 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.607271 (+ 65us) server_negotiation.cc:407] Connection header received
0620 02:36:13.609645 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.609679 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.610145 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.610617 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:13.612246 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.613688 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.614994 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.615728 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.615963 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.620660 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.620671 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.620686 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.620742 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.625149 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.626017 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.626034 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.626042 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.626254 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.626650 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.626656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.626666 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.627345 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.627631 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.628127 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.628650 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":522,"threads_started":1}
I20260620 02:36:13.634466 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.618528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.0.97.62:36009)
0620 02:36:13.618760 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.618768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.619488 (+ 720us) server_negotiation.cc:407] Connection header received
0620 02:36:13.619708 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.619714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.619798 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.619943 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:13.621551 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.622342 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.624831 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.625270 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.625435 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.629159 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.629165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.629168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.629204 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.631896 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.632787 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.632791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.632794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.632912 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.633266 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.633269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.633271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.633529 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.633642 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.634043 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.634237 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:36:13.634491 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.618329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49154)
0620 02:36:13.619317 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.619364 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:13.619559 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:13.620034 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:13.620040 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:13.620950 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:13.621343 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.621364 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.622725 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.622737 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.624631 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.624649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.625457 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.625461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.625583 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.626303 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:13.626321 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:13.628922 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:13.632030 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.632039 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.632046 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.632692 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.633014 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.633018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.633020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.633163 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.633645 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:13.633650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:13.633797 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 02:36:13.633957 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 02:36:13.634206 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":499,"threads_started":1}
I20260620 02:36:13.636068 7274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:36:13 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:36:13.650321 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.637310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.0.97.61:44071)
0620 02:36:13.637482 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.637487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.637503 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:36:13.637551 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.637554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.637615 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.637697 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:13.638392 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.639180 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.640586 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.641451 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.641568 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.645043 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.645052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.645056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.645091 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.647595 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.648370 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.648375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.648379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.648514 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.648928 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.648931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.648933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.649224 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.649375 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.649734 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.649961 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:36:13.650312 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.636983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44071 (local address 127.0.0.1:35232)
0620 02:36:13.637219 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.637241 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:13.637345 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:13.637735 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:13.637738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:13.638060 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:13.638230 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.638239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.639398 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.639403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.640426 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.640438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.641637 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.641643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.641793 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.642553 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:13.642572 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:13.644808 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:13.647755 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.647761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.647765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.648238 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.648633 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.648638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.648642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.648784 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.649342 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:13.649347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:13.649495 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 02:36:13.649636 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 02:36:13.649964 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 02:36:13.652019 7274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:36:13 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:36:13.664893 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.652769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42509 (local address 127.0.0.1:53126)
0620 02:36:13.652987 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.653003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:13.653092 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:13.653356 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:13.653359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:13.653717 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:13.653863 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.653871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.654880 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.654884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.655722 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.655733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.656540 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.656545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.656700 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.657274 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:13.657292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:13.659358 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:13.662457 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.662461 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.662464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.662832 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.663306 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.663310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.663312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.663470 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.664070 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:13.664082 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:13.664213 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 02:36:13.664382 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 02:36:13.664602 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 02:36:13.665146 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.652786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.0.97.60:42509)
0620 02:36:13.652961 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.652965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.653076 (+ 111us) server_negotiation.cc:407] Connection header received
0620 02:36:13.653171 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.653175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.653235 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.653321 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:13.653986 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.654762 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.655931 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.656378 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.656481 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.659550 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.659556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.659559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.659591 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.662299 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.662990 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.662995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.662998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.663149 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.663580 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.663584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.663586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.663964 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.664095 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.664469 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.664750 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":64}
I20260620 02:36:13.671012 7274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: Bootstrap starting.
I20260620 02:36:13.674443 7274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:13.675504 7274 log.cc:824] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:13.678551 7274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: No bootstrap required, opened a new log
I20260620 02:36:13.686587 7274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } }
I20260620 02:36:13.686959 7274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:13.687036 7274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809c672b0cdc4956ada9756e748673bc, State: Initialized, Role: FOLLOWER
I20260620 02:36:13.687562 7274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } }
I20260620 02:36:13.688552 7280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } } }
I20260620 02:36:13.688917 7280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:36:13.689668 7274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:36:13.700650 7291 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:36:13.700850 7291 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:36:13.706053 7118 master_runner.cc:428] Detected that this master 809c672b0cdc4956ada9756e748673bc is joining an existing cluster
I20260620 02:36:13.706209 7118 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:42509
I20260620 02:36:13.737301 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 6617
I20260620 02:36:13.744642 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.712072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.0.97.62:36009)
0620 02:36:13.712700 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.712706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.720654 (+ 7948us) server_negotiation.cc:407] Connection header received
0620 02:36:13.720875 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.720880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.720964 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.721060 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:13.723364 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.724705 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.727407 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.728116 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.728232 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.733433 (+ 5201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.733442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.733446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.733489 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.737958 (+ 4469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.738909 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.738916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.738919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.739117 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.739573 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.739580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.739586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.740023 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.740149 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.743595 (+ 3446us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.744228 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478}
I20260620 02:36:13.746209 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.712507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49156)
0620 02:36:13.720464 (+ 7957us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.720495 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:13.720732 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:13.721281 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:13.721286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:13.722066 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:13.723091 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.723115 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.725259 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.725265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.727202 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.727217 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.728560 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.728566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.728797 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.730358 (+ 1561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:13.730379 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:13.733093 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:13.738148 (+ 5055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.738156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.738162 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.738686 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.739247 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.739254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.739257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.739434 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.740232 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:13.740238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:13.740482 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 02:36:13.745259 (+ 4777us) client_negotiation.cc:240] Negotiation successful
0620 02:36:13.745630 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7777,"spinlock_wait_cycles":58112,"thread_start_us":10298,"threads_started":1}
I20260620 02:36:13.748378 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.712291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.0.97.61:44071)
0620 02:36:13.712987 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.712997 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.723712 (+ 10715us) server_negotiation.cc:407] Connection header received
0620 02:36:13.724041 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.724047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.724137 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.724211 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:13.726350 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.727524 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.730534 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.731213 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.731361 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.734882 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.734890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.734895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.734939 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.738729 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.739858 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.739864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.739868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.740020 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.740487 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.740492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.740495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.740905 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.741034 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.747679 (+ 6645us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.748023 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440}
I20260620 02:36:13.749486 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.713600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44071 (local address 127.0.0.1:35234)
0620 02:36:13.723587 (+ 9987us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.723610 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:13.723869 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:13.724427 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:13.724433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:13.725656 (+ 1223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:13.726142 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.726156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.727934 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.727939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.730350 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.730369 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.731441 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.731448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.731662 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.732400 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:13.732423 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:13.734592 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:13.738910 (+ 4318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.738915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.738919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.739681 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.740174 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.740178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.740182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.740339 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.741110 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:13.741115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:13.741343 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 02:36:13.748636 (+ 7293us) client_negotiation.cc:240] Negotiation successful
0620 02:36:13.748962 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9843,"mutex_wait_us":3,"thread_start_us":7427,"threads_started":1}
I20260620 02:36:13.751643 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.712484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.0.97.60:42509)
0620 02:36:13.713236 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.713244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.721011 (+ 7767us) server_negotiation.cc:407] Connection header received
0620 02:36:13.721565 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.721578 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.721669 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.721761 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:13.724100 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.725572 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.727635 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.728479 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.728571 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.731787 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.731799 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.731806 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.731898 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.744556 (+ 12658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.745443 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.745448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.745452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.745615 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.746016 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.746021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.746024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.746439 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.746588 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.751018 (+ 4430us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.751307 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558}
I20260620 02:36:13.753278 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.712646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42509 (local address 127.0.0.1:53128)
0620 02:36:13.720879 (+ 8233us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.720907 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:13.721384 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:13.722026 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:13.722033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:13.723541 (+ 1508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:13.723926 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.723942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.725799 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.725805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.727506 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.727518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.728886 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.728897 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.729090 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.730295 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:13.730333 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:13.731591 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:13.744746 (+ 13155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.744754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.744758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.745320 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.745763 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.745768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.745772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.745913 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.746676 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:13.746683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:13.747209 (+ 526us) client_negotiation.cc:769] Sending connection context
0620 02:36:13.751895 (+ 4686us) client_negotiation.cc:240] Negotiation successful
0620 02:36:13.752334 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8020,"thread_start_us":7447,"threads_started":1}
I20260620 02:36:13.754876 7280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:36:13.755246 7280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } }
I20260620 02:36:13.756980 7280 leader_election.cc:288] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009), f48ad3e294f44c9ba44e8baa08401195 (127.0.97.61:44071)
I20260620 02:36:13.757891 6566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "809c672b0cdc4956ada9756e748673bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b53dba5fad064c26a173bd4691e8a17c" is_pre_election: true
I20260620 02:36:13.759958 7166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "809c672b0cdc4956ada9756e748673bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48ad3e294f44c9ba44e8baa08401195" is_pre_election: true
I20260620 02:36:13.760866 7226 leader_election.cc:302] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 809c672b0cdc4956ada9756e748673bc; no voters: b53dba5fad064c26a173bd4691e8a17c, f48ad3e294f44c9ba44e8baa08401195
Jun 20 02:36:13 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
Jun 20 02:36:13 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:36:13.774451 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.739219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.0.97.1:42313)
0620 02:36:13.753495 (+ 14276us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.753506 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.753689 (+ 183us) server_negotiation.cc:407] Connection header received
0620 02:36:13.753794 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.753800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.754151 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.754395 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:13.756174 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.758238 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.760268 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.761186 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.765768 (+ 4582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.767735 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.767749 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.767766 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.767807 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.771334 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.772199 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.772207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.772209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.772349 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.772726 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.772731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.772736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.773106 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.773429 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.773759 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.774044 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14057,"thread_start_us":15377,"threads_started":1}
I20260620 02:36:13.775784 6740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:13.776018 7280 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 02:36:13.783743 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.754011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.0.97.60:42509)
0620 02:36:13.756021 (+ 2010us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.756029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.756063 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:36:13.756166 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.756172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.756267 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.757809 (+ 1542us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:13.757990 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.760941 (+ 2951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.763435 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.764241 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.764402 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.769606 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.769615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.769621 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.769673 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.780045 (+ 10372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.781278 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.781285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.781289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.781443 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.781923 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.781929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.781932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.782321 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.782450 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.783077 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.783344 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1888}
I20260620 02:36:13.784716 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.753601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42509 (local address 127.0.0.1:53138)
0620 02:36:13.754059 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.754079 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:13.754213 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:13.756471 (+ 2258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:13.756476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:13.757118 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:13.757543 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.757561 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.761192 (+ 3631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.761197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.763171 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.763192 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.764451 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.764458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.764669 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.765407 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:13.765448 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:13.768790 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:13.780249 (+ 11459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.780259 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.780266 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.781114 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.781601 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.781607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.781611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.781771 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.782547 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:13.782552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:13.782744 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 02:36:13.784012 (+ 1268us) client_negotiation.cc:240] Negotiation successful
0620 02:36:13.784351 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303}
I20260620 02:36:13.786346 6556 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:42509: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "809c672b0cdc4956ada9756e748673bc"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 42509
}
attrs {
promote: true
}
}
dest_uuid: "b53dba5fad064c26a173bd4691e8a17c"
cas_config_opid_index: 5
I20260620 02:36:13.787501 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } }
I20260620 02:36:13.789577 7117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 809c672b0cdc4956ada9756e748673bc (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:13.790345 7166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 LEARNER]: Refusing update from remote peer b53dba5fad064c26a173bd4691e8a17c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:36:13.793853 7117 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:42509
I20260620 02:36:13.794598 7118 master.cc:537] Master@127.0.97.60:42509 shutting down...
I20260620 02:36:13.795549 7249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:13.798023 7302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:36:13.799574 7249 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 FOLLOWER]: Refusing update from remote peer b53dba5fad064c26a173bd4691e8a17c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:36:13.811112 7303 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:13.812287 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:13.812669 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:13.814766 7118 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:13.815291 7118 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:13.815575 7118 tablet_replica.cc:330] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: stopping tablet replica
I20260620 02:36:13.816256 7166 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 809c672b0cdc4956ada9756e748673bc (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:13.822770 7118 master.cc:552] Master@127.0.97.60:42509 shutdown complete.
I20260620 02:36:13.822942 7118 master_runner.cc:305] Clearing existing system tablet
I20260620 02:36:13.823402 7118 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:13.823539 7118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:13.823726 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.823238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42509 (local address 127.0.0.1:53148)
0620 02:36:13.823403 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.823529 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 02:36:13.825039 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: true } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:13.825459 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:36:13.826737 7302 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c: attempting to promote NON_VOTER f48ad3e294f44c9ba44e8baa08401195 to VOTER
I20260620 02:36:13.828058 7302 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } }
I20260620 02:36:13.831059 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 LEARNER]: Refusing update from remote peer b53dba5fad064c26a173bd4691e8a17c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 02:36:13.831946 7117 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:36:13.833684 7118 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.007s
I20260620 02:36:13.834785 7117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f48ad3e294f44c9ba44e8baa08401195 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:13.837968 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f48ad3e294f44c9ba44e8baa08401195 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:13.843887 7309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:13.844376 7118 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:13.844852 7118 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.009s
I20260620 02:36:13.845008 7118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/wal
uuid: "809c672b0cdc4956ada9756e748673bc"
format_stamp: "Formatted at 2026-06-20 02:36:13 on dist-test-slave-rpg6"
server_key: "77bf399fcced37806980944310991b9b"
server_key_iv: "2e3b7f18b1598c7b926e13164dc23425"
server_key_version: "encryptionkey@0"
I20260620 02:36:13.845788 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:13.846061 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:36:13.846432 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:13.846808 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:13.846608 7305 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:13.846992 7305 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:13.848191 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.847737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42509 (local address 127.0.0.1:53160)
0620 02:36:13.847926 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.848024 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 02:36:13.849586 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.849190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42509 (local address 127.0.0.1:53170)
0620 02:36:13.849322 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.849398 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 02:36:13.849839 6540 consensus_peers.cc:487] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c -> Peer 809c672b0cdc4956ada9756e748673bc (127.0.97.60:42509): Couldn't send request to peer 809c672b0cdc4956ada9756e748673bc. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 02:36:13.851774 7118 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:13.859416 7118 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:36:13.859756 7118 log.cc:1196] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 02:36:13.860395 7118 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: Deleting consensus metadata
I20260620 02:36:13.860710 7118 master_runner.cc:315] Copying system tablet from 127.0.97.62:36009
I20260620 02:36:13.863412 7118 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:36009
I20260620 02:36:13.884169 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.864399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49170 (local address 127.0.97.62:36009)
0620 02:36:13.864611 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:13.864615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:13.868489 (+ 3874us) server_negotiation.cc:407] Connection header received
0620 02:36:13.868696 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:13.868702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:13.868789 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:13.868903 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:13.870526 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.871710 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.873665 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.874326 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.874464 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.877128 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:13.877135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:13.877140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:13.877181 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:13.880923 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.881903 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.881910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.881914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.882106 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:13.882558 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:13.882563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:13.882573 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:13.882919 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:13.883076 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:13.883542 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 02:36:13.883789 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:36:13.884423 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:13.864339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49170)
0620 02:36:13.868319 (+ 3980us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:13.868349 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:13.868538 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:13.868965 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:13.868970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:13.870019 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:13.870305 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.870320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.871914 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.871921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.873458 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:13.873476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:13.874509 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:13.874521 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:13.874723 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:13.875535 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:13.875558 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:13.876862 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:13.881147 (+ 4285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.881154 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.881159 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.881713 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.882235 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:13.882240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:13.882244 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:13.882429 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:13.883046 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:13.883052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:13.883261 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 02:36:13.883404 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 02:36:13.883757 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3781,"thread_start_us":3647,"threads_started":1}
I20260620 02:36:13.885108 6576 tablet_copy_service.cc:140] P b53dba5fad064c26a173bd4691e8a17c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 809c672b0cdc4956ada9756e748673bc ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:49170)
I20260620 02:36:13.885294 6576 tablet_copy_service.cc:161] P b53dba5fad064c26a173bd4691e8a17c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 809c672b0cdc4956ada9756e748673bc at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:49170: session id = 809c672b0cdc4956ada9756e748673bc-00000000000000000000000000000000
I20260620 02:36:13.888203 6576 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:36:13.890157 7118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:36:13.896173 7118 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: tablet copy: Starting download of 0 data blocks...
I20260620 02:36:13.896498 7118 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: tablet copy: Starting download of 1 WAL segments...
I20260620 02:36:13.898908 7118 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:36:13.902990 6576 tablet_copy_service.cc:342] P b53dba5fad064c26a173bd4691e8a17c: Request end of tablet copy session 809c672b0cdc4956ada9756e748673bc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:49170
I20260620 02:36:13.903168 6576 tablet_copy_service.cc:433] P b53dba5fad064c26a173bd4691e8a17c: ending tablet copy session 809c672b0cdc4956ada9756e748673bc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 809c672b0cdc4956ada9756e748673bc
I20260620 02:36:13.914573 7118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:13.917533 7316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:13.927227 7319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:13.927812 7317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:13.928150 7118 server_base.cc:1034] running on GCE node
I20260620 02:36:13.928603 7118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:13.929423 7118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:13.930702 7118 hybrid_clock.cc:648] HybridClock initialized: now 1781922973930656 us; error 38 us; skew 500 ppm
I20260620 02:36:13.931031 7118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:13 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:13.933158 7118 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 02:36:13.934544 7118 webserver.cc:460] Webserver started at http://127.0.97.60:46217/ using document root <none> and password file <none>
I20260620 02:36:13.934980 7118 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:13.935098 7118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:13.939034 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 6617
I20260620 02:36:13.940438 7118 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:36:13.942252 7326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:13.942709 7118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 02:36:13.942876 7118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/wal
uuid: "809c672b0cdc4956ada9756e748673bc"
format_stamp: "Formatted at 2026-06-20 02:36:13 on dist-test-slave-rpg6"
server_key: "77bf399fcced37806980944310991b9b"
server_key_iv: "2e3b7f18b1598c7b926e13164dc23425"
server_key_version: "encryptionkey@0"
I20260620 02:36:13.943073 7118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:13.964982 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42313
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36009
--webserver_port=46835
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:36009,127.0.97.61:44071
--tserver_master_addrs=127.0.97.62:36009,127.0.97.61:44071,127.0.97.60:42509 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:36:13.977839 7118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:13.982009 7118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:14.020373 7118 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:42509
I20260620 02:36:14.020421 7378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:42509 every 8 connection(s)
I20260620 02:36:14.021509 7118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:36:14.027158 7379 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:36:14.031630 7379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: Bootstrap starting.
I20260620 02:36:14.056239 7379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:36:14.056928 7379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: Bootstrap complete.
I20260620 02:36:14.058233 7379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } }
I20260620 02:36:14.058593 7379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 809c672b0cdc4956ada9756e748673bc, State: Initialized, Role: LEARNER
I20260620 02:36:14.058920 7379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } }
I20260620 02:36:14.060765 7379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:14.060518 7382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:14.061089 7382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: This master's current role is: LEARNER
I20260620 02:36:14.073207 7393 catalog_manager.cc:1259] Loaded cluster ID: 909a3d673ad54206ac43ca677703694b
I20260620 02:36:14.073344 7393 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: loading cluster ID for follower catalog manager: success
I20260620 02:36:14.077286 7393 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: acquiring CA information for follower catalog manager: success
I20260620 02:36:14.080008 7393 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:36:14.362023 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:14.354113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42509 (local address 127.0.0.1:53176)
0620 02:36:14.354621 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:14.354647 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:14.354816 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:14.355254 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:14.355259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:14.355284 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:14.355489 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:14.355498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.357068 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.357077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:14.358904 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:14.358925 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.361177 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.361187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:14.361400 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:14.361437 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:36:14.361536 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 02:36:14.361711 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":294,"threads_started":1}
I20260620 02:36:14.362852 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:14.354270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.0.97.60:42509)
0620 02:36:14.354935 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:14.354941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:14.354968 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:36:14.355028 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:14.355032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:14.355098 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:14.355240 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:14.355655 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.356895 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.359119 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.360677 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.360867 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:14.362422 (+ 1555us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:14.362480 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 02:36:14.362612 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":375,"threads_started":1}
I20260620 02:36:14.363770 7354 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 02:36:14.364444 7382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53dba5fad064c26a173bd4691e8a17c. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:14.364768 7382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: This master's current role is: LEARNER
W20260620 02:36:14.382262 7331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:14.382580 7331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:14.382688 7331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:14.390442 7331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:14.390588 7331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:14.390766 7331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:36:14.403040 7331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46835
--tserver_master_addrs=127.0.97.62:36009,127.0.97.61:44071,127.0.97.60:42509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:14.404603 7331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:14.406029 7331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:14.422251 7400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:14.422422 7401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:14.422322 7403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:14.423718 7331 server_base.cc:1034] running on GCE node
I20260620 02:36:14.424547 7331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:14.425624 7331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:14.426931 7331 hybrid_clock.cc:648] HybridClock initialized: now 1781922974426872 us; error 57 us; skew 500 ppm
I20260620 02:36:14.427363 7331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:14 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922974, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:14.430962 7331 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:36:14.432802 7331 webserver.cc:460] Webserver started at http://127.0.97.1:46835/ using document root <none> and password file <none>
I20260620 02:36:14.433554 7331 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:14.433722 7331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:14.439800 7331 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:36:14.442997 7410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:14.444741 7331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:36:14.444933 7331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "c62ac2bd7747422ebb0a58bcb681f6e1"
format_stamp: "Formatted at 2026-06-20 02:36:11 on dist-test-slave-rpg6"
server_key: "c18b0bc5dd7ab7780e95d62ddbbaf05a"
server_key_iv: "4521b32a07503f10d259f10709c9048c"
server_key_version: "encryptionkey@0"
I20260620 02:36:14.446491 7331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:14.468035 7331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:14.474153 7331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:14.474468 7331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:14.475605 7331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:14.477531 7331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:14.477633 7331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:14.477711 7331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:14.477769 7331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:14.518062 7331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42313
I20260620 02:36:14.518117 7522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42313 every 8 connection(s)
I20260620 02:36:14.519589 7331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:36:14.520732 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 7331
I20260620 02:36:14.521128 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 6751
Jun 20 02:36:14 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 02:36:14.546638 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:14.522179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.0.97.2:41019)
0620 02:36:14.523301 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:14.523312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:14.524030 (+ 718us) server_negotiation.cc:407] Connection header received
0620 02:36:14.524386 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:14.524399 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:14.524789 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:14.524977 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:14.528060 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.530030 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.531618 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.532543 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.532674 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:14.539199 (+ 6525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:14.539214 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:14.539228 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:14.539267 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:14.542388 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:14.543543 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:14.543560 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:14.543574 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:14.543816 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:14.544666 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:14.544672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:14.544675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:14.545115 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:14.545417 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:14.545914 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 02:36:14.546198 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":943,"threads_started":1}
I20260620 02:36:14.548599 6874 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 20 02:36:14 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922974, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 02:36:14 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922974, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 02:36:14 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922974, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:36:14.565860 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:14.524129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40885 (local address 127.0.97.62:36009)
0620 02:36:14.536017 (+ 11888us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:14.536025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:14.536046 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:36:14.537523 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:14.537530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:14.537643 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:14.537737 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:14.542556 (+ 4819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.543667 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.547409 (+ 3742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.548140 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.548274 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:14.554694 (+ 6420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:14.554705 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:14.554712 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:14.554772 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:14.559703 (+ 4931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:14.561087 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:14.561094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:14.561099 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:14.561266 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:14.562350 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:14.562356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:14.562360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:14.562745 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:14.562927 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:14.564777 (+ 1850us) server_negotiation.cc:299] Negotiation successful
0620 02:36:14.565108 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11544,"thread_start_us":11411,"threads_started":1}
I20260620 02:36:14.567991 7533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:14.524182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.1:40885)
0620 02:36:14.532311 (+ 8129us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:14.532374 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:14.537083 (+ 4709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:14.538524 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:14.538538 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:14.540999 (+ 2461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:14.542237 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:14.542296 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.543855 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.543865 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:14.546568 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:14.546591 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.548591 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.548599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:14.548949 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:14.549923 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:14.549957 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:14.554354 (+ 4397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:14.560265 (+ 5911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:14.560276 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:14.560297 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:14.560955 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:14.561851 (+ 896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:14.561857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:14.561862 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:14.562160 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:14.563014 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:14.563025 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:14.563556 (+ 531us) client_negotiation.cc:769] Sending connection context
0620 02:36:14.563947 (+ 391us) client_negotiation.cc:240] Negotiation successful
0620 02:36:14.566216 (+ 2269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7736,"thread_start_us":22596,"threads_started":1}
I20260620 02:36:14.568318 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:14.523807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54215 (local address 127.0.97.61:44071)
0620 02:36:14.536425 (+ 12618us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:14.536432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:14.547358 (+ 10926us) server_negotiation.cc:407] Connection header received
0620 02:36:14.547612 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:14.547618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:14.547713 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:14.547792 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:14.549341 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.550667 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.553649 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.554285 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.554387 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:14.560315 (+ 5928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:14.560321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:14.560324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:14.560359 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:14.564059 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:14.564956 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:14.564962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:14.564969 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:14.565134 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:14.566297 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:14.566302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:14.566305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:14.566654 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:14.566812 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:14.567441 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 02:36:14.567697 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12488,"thread_start_us":12343,"threads_started":1}
I20260620 02:36:14.569446 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:14.522918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41507 (local address 127.0.97.60:42509)
0620 02:36:14.523115 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:14.523121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:14.535371 (+ 12250us) server_negotiation.cc:407] Connection header received
0620 02:36:14.537605 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:14.537615 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:14.537736 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:14.537829 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:14.547344 (+ 9515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.548434 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.551747 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.552695 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.552819 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:14.561907 (+ 9088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:14.561915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:14.561922 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:14.561969 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:14.566021 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:14.566957 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:14.566963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:14.566967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:14.567121 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:14.567531 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:14.567536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:14.567550 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:14.567933 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:14.568060 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:14.568601 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 02:36:14.569104 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 02:36:14.569542 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:14.527440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44071 (local address 127.0.97.1:54215)
0620 02:36:14.547254 (+ 19814us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:14.547275 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:14.547467 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:14.548100 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:14.548105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:14.548723 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:14.549141 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:14.549153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.550988 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.550992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:14.553458 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:14.553473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.554671 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.554676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:14.554805 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:14.555558 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:14.555584 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:14.560083 (+ 4499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:14.564231 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:14.564239 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:14.564244 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:14.564806 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:14.565288 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:14.565295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:14.565299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:14.565868 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:14.566900 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:14.566908 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:14.567151 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 02:36:14.568800 (+ 1649us) client_negotiation.cc:240] Negotiation successful
0620 02:36:14.569115 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19658,"mutex_wait_us":83,"thread_start_us":4537,"threads_started":1}
I20260620 02:36:14.569962 7524 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36009
I20260620 02:36:14.570312 7524 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:14.570495 7525 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44071
I20260620 02:36:14.570642 7525 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:14.571235 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:14.524609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42509 (local address 127.0.97.1:41507)
0620 02:36:14.535237 (+ 10628us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:14.535266 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:14.537090 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:14.538524 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:14.538540 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:14.544521 (+ 5981us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:14.547123 (+ 2602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:14.547137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.548640 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.548644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:14.551507 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:14.551528 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:14.553188 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:14.553195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:14.553433 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:14.557948 (+ 4515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:14.557982 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:14.561666 (+ 3684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:14.566174 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:14.566179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:14.566183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:14.566814 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:14.567269 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:14.567275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:14.567278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:14.567413 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:14.568157 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:14.568163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:14.568302 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 02:36:14.569286 (+ 984us) client_negotiation.cc:240] Negotiation successful
0620 02:36:14.570345 (+ 1059us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10464,"mutex_wait_us":1614,"thread_start_us":10246,"threads_started":1}
I20260620 02:36:14.571426 7525 heartbeater.cc:511] Master 127.0.97.61:44071 requested a full tablet report, sending...
I20260620 02:36:14.571555 7524 heartbeater.cc:511] Master 127.0.97.62:36009 requested a full tablet report, sending...
I20260620 02:36:14.572233 7523 heartbeater.cc:347] Connected to a master server at 127.0.97.60:42509
I20260620 02:36:14.572357 7523 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:14.572794 7523 heartbeater.cc:511] Master 127.0.97.60:42509 requested a full tablet report, sending...
I20260620 02:36:14.573334 6556 ts_manager.cc:194] Re-registered known tserver with Master: c62ac2bd7747422ebb0a58bcb681f6e1 (127.0.97.1:42313)
I20260620 02:36:14.573907 7156 ts_manager.cc:194] Registered new tserver with Master: c62ac2bd7747422ebb0a58bcb681f6e1 (127.0.97.1:42313)
I20260620 02:36:14.574430 7344 ts_manager.cc:194] Registered new tserver with Master: c62ac2bd7747422ebb0a58bcb681f6e1 (127.0.97.1:42313)
I20260620 02:36:14.574456 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40885
I20260620 02:36:14.646709 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 6751
I20260620 02:36:14.663105 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41019
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36009
--webserver_port=39777
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:36009,127.0.97.61:44071
--tserver_master_addrs=127.0.97.62:36009,127.0.97.61:44071,127.0.97.60:42509 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:36:14.740617 7305 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: true } } }
I20260620 02:36:14.741039 7305 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:14.743090 7305 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c: attempting to promote NON_VOTER 809c672b0cdc4956ada9756e748673bc to VOTER
I20260620 02:36:14.743755 7305 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } }
I20260620 02:36:14.745829 7354 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 LEARNER]: Refusing update from remote peer b53dba5fad064c26a173bd4691e8a17c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 02:36:14.745898 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Refusing update from remote peer b53dba5fad064c26a173bd4691e8a17c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 02:36:14.746536 7537 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 02:36:14.747059 7536 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 02:36:14.749559 7537 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 809c672b0cdc4956ada9756e748673bc (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:14.750329 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 809c672b0cdc4956ada9756e748673bc (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:14.753875 7305 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:14.754176 7305 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:14.754577 7305 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:14.754824 7305 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:14.758476 7540 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:14.759135 7540 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:36:14.761348 7536 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:14.761621 7536 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:14.762004 7354 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 809c672b0cdc4956ada9756e748673bc (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:14.766355 7382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b53dba5fad064c26a173bd4691e8a17c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:14.766650 7382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: This master's current role is: FOLLOWER
W20260620 02:36:15.027101 7535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:15.027401 7535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:15.027568 7535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:15.035434 7535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:15.035570 7535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:15.035763 7535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:36:15.046737 7535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39777
--tserver_master_addrs=127.0.97.62:36009,127.0.97.61:44071,127.0.97.60:42509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:15.048367 7535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:15.050706 7535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:15.069830 7545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:15.069972 7546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:15.070232 7548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:15.070905 7535 server_base.cc:1034] running on GCE node
I20260620 02:36:15.071916 7535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:15.073158 7535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:15.074456 7535 hybrid_clock.cc:648] HybridClock initialized: now 1781922975074389 us; error 62 us; skew 500 ppm
I20260620 02:36:15.074880 7535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:15 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:15.079274 7535 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:36:15.081527 7535 webserver.cc:460] Webserver started at http://127.0.97.2:39777/ using document root <none> and password file <none>
I20260620 02:36:15.082338 7535 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:15.082509 7535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:15.088766 7535 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:36:15.092017 7555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:15.093676 7535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:36:15.093905 7535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "94cd1db181e54aeca840315161cb9e67"
format_stamp: "Formatted at 2026-06-20 02:36:11 on dist-test-slave-rpg6"
server_key: "c32df84751588665fa5d889d503d3c4b"
server_key_iv: "1277c7b58c54e2058819fdfc1bcaf795"
server_key_version: "encryptionkey@0"
I20260620 02:36:15.095103 7535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:15.140851 7535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:15.145787 7535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:15.146103 7535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:15.147162 7535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:15.148769 7535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:15.148892 7535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:15.149032 7535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:15.149134 7535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:15.186221 7535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41019
I20260620 02:36:15.186321 7667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41019 every 8 connection(s)
I20260620 02:36:15.187714 7535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:36:15.198993 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 7535
I20260620 02:36:15.199286 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 6885
Jun 20 02:36:15 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 02:36:15 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 02:36:15 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:15.217852 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.191480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56801 (local address 127.0.97.60:42509)
0620 02:36:15.192574 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.192582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.194583 (+ 2001us) server_negotiation.cc:407] Connection header received
0620 02:36:15.196569 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.196576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.196662 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.196762 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:15.199170 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.200271 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.203111 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.203777 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.203958 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.209861 (+ 5903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:15.209873 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:15.209881 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:15.209944 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:15.213638 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.214886 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.214893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.214897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.215047 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.215499 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.215507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.215510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.215938 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:15.216089 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.217212 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.217482 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":859,"threads_started":1}
I20260620 02:36:15.219149 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.192102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42509 (local address 127.0.97.2:56801)
0620 02:36:15.194451 (+ 2349us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:15.194471 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:15.196145 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:15.196956 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:15.196961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:15.198129 (+ 1168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:15.198950 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.198968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.201637 (+ 2669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.201641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.202933 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.202947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.204104 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.204115 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.204529 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.205541 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:15.205572 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:15.209540 (+ 3968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:15.213856 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.213867 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.213890 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.214762 (+ 872us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.215178 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.215183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.215186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.215369 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.216175 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:15.216186 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:15.216695 (+ 509us) client_negotiation.cc:769] Sending connection context
0620 02:36:15.217084 (+ 389us) client_negotiation.cc:240] Negotiation successful
0620 02:36:15.217638 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2097,"mutex_wait_us":161,"spinlock_wait_cycles":17152,"thread_start_us":489,"threads_started":1}
I20260620 02:36:15.219599 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.191258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50605 (local address 127.0.97.61:44071)
0620 02:36:15.191731 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.191781 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.193609 (+ 1828us) server_negotiation.cc:407] Connection header received
0620 02:36:15.196438 (+ 2829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.196445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.196554 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.196695 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:15.199169 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.200240 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.203301 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.203989 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.204108 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.209264 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:15.209272 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:15.209277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:15.209338 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:15.215347 (+ 6009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.216904 (+ 1557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.216910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.216916 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.217071 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.217533 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.217538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.217542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.217918 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:15.218044 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.218699 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.218943 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":400,"threads_started":1}
I20260620 02:36:15.220675 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.192102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44071 (local address 127.0.97.2:50605)
0620 02:36:15.193358 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:15.193437 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:15.196110 (+ 2673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:15.196897 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:15.196906 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:15.197845 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:15.198921 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.198949 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.200489 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.200498 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.203080 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.203101 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.204326 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.204331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.204521 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.205385 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:15.205424 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:15.208899 (+ 3475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:15.215968 (+ 7069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.216010 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.216015 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.216712 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.217229 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.217234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.217237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.217396 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.218163 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:15.218170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:15.218373 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 02:36:15.218537 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 02:36:15.218840 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":133,"spinlock_wait_cycles":4992,"thread_start_us":1823,"threads_started":1}
I20260620 02:36:15.221712 7668 heartbeater.cc:347] Connected to a master server at 127.0.97.60:42509
I20260620 02:36:15.221721 7670 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44071
I20260620 02:36:15.222139 7668 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:15.222143 7670 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:15.223285 7670 heartbeater.cc:511] Master 127.0.97.61:44071 requested a full tablet report, sending...
I20260620 02:36:15.223392 7668 heartbeater.cc:511] Master 127.0.97.60:42509 requested a full tablet report, sending...
I20260620 02:36:15.225359 7344 ts_manager.cc:194] Registered new tserver with Master: 94cd1db181e54aeca840315161cb9e67 (127.0.97.2:41019)
I20260620 02:36:15.225369 7156 ts_manager.cc:194] Registered new tserver with Master: 94cd1db181e54aeca840315161cb9e67 (127.0.97.2:41019)
I20260620 02:36:15.225471 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.192286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.2:43769)
0620 02:36:15.194991 (+ 2705us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:15.195004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:15.196110 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:15.197236 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:15.197240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:15.198292 (+ 1052us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:15.198920 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.198950 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.206349 (+ 7399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.206357 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.209683 (+ 3326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.209701 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.211587 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.211595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.211780 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.212383 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:15.212405 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:15.216343 (+ 3938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:15.222056 (+ 5713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.222063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.222070 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.222831 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.223560 (+ 729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.223566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.223569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.223761 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.224410 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:15.224415 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:15.224579 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 02:36:15.224733 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 02:36:15.225043 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2607,"mutex_wait_us":95,"spinlock_wait_cycles":82688,"thread_start_us":2403,"threads_started":1}
I20260620 02:36:15.227943 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.191546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43769 (local address 127.0.97.62:36009)
0620 02:36:15.192246 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.192252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.195105 (+ 2853us) server_negotiation.cc:407] Connection header received
0620 02:36:15.196959 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.196964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.197065 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.197184 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:15.200460 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.201450 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.210652 (+ 9202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.211390 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.211520 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.217338 (+ 5818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:15.217345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:15.217350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:15.217393 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:15.221890 (+ 4497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.223213 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.223219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.223223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.223395 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.223914 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.223920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.223923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.224274 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:15.224426 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.227255 (+ 2829us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.227448 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":446,"threads_started":1}
I20260620 02:36:15.228847 7669 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36009
I20260620 02:36:15.229002 7669 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:15.229463 7669 heartbeater.cc:511] Master 127.0.97.62:36009 requested a full tablet report, sending...
I20260620 02:36:15.230577 6556 ts_manager.cc:194] Re-registered known tserver with Master: 94cd1db181e54aeca840315161cb9e67 (127.0.97.2:41019)
Jun 20 02:36:15 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 02:36:15.231770 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43769
I20260620 02:36:15.241149 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.200923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.0.97.3:34605)
0620 02:36:15.201538 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.201548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.211321 (+ 9773us) server_negotiation.cc:407] Connection header received
0620 02:36:15.212260 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.212274 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.212619 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.212798 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:15.214850 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.222988 (+ 8138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.226097 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.226865 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.226978 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.232910 (+ 5932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:15.232934 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:15.232955 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:15.233002 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:15.236703 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.237978 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.237986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.237989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.238201 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.239248 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.239253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.239255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.239758 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:15.240136 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.240496 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.240772 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":275,"threads_started":1}
I20260620 02:36:15.243112 7008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:15.331094 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 6885
I20260620 02:36:15.348009 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34605
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36009
--webserver_port=37691
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:36009,127.0.97.61:44071
--tserver_master_addrs=127.0.97.62:36009,127.0.97.61:44071,127.0.97.60:42509 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:36:15.577054 7524 heartbeater.cc:503] Master 127.0.97.62:36009 was elected leader, sending a full tablet report...
W20260620 02:36:15.710964 7682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:15.711285 7682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:15.711385 7682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:15.719153 7682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:15.719291 7682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:15.719408 7682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:36:15.730422 7682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37691
--tserver_master_addrs=127.0.97.62:36009,127.0.97.61:44071,127.0.97.60:42509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:15.731812 7682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:15.733282 7682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:15.749099 7687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:15.749117 7688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:15.749183 7690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:15.749990 7682 server_base.cc:1034] running on GCE node
I20260620 02:36:15.750726 7682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:15.752211 7682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:15.753507 7682 hybrid_clock.cc:648] HybridClock initialized: now 1781922975753431 us; error 68 us; skew 500 ppm
I20260620 02:36:15.753895 7682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:15 dist-test-slave-rpg6 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:15.757609 7682 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:36:15.759388 7682 webserver.cc:460] Webserver started at http://127.0.97.3:37691/ using document root <none> and password file <none>
I20260620 02:36:15.760445 7682 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:15.760636 7682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:15.767083 7682 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 02:36:15.771308 7697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:15.773373 7682 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:36:15.773612 7682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "056f42312e944df685297ea3645a0ede"
format_stamp: "Formatted at 2026-06-20 02:36:12 on dist-test-slave-rpg6"
server_key: "4e55a485b99035184ff0ccc6128cee1e"
server_key_iv: "8b5040725a29ed891d040d8f6e95e433"
server_key_version: "encryptionkey@0"
I20260620 02:36:15.774735 7682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:15.790570 7682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:15.796015 7682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:15.796345 7682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:15.797542 7682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:15.799261 7682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:15.799352 7682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:15.799471 7682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:15.799563 7682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:15.836526 7682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34605
I20260620 02:36:15.836719 7809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34605 every 8 connection(s)
I20260620 02:36:15.838095 7682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:36:15.843056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 7682
Jun 20 02:36:15 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 02:36:15 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 02:36:15 dist-test-slave-rpg6 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922975, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:36:15.884001 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.841432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43929 (local address 127.0.97.61:44071)
0620 02:36:15.854467 (+ 13035us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.854473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.854493 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:36:15.854548 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.854553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.854630 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.856052 (+ 1422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:15.856259 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.857446 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.861179 (+ 3733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.862178 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.862322 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.871294 (+ 8972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:15.871306 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:15.871315 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:15.871372 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:15.876807 (+ 5435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.880546 (+ 3739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.880553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.880557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.880756 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.881362 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.881367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.881370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.881725 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:15.881875 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.883268 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.883509 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12908,"thread_start_us":12789,"threads_started":1}
I20260620 02:36:15.885548 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.842403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.3:36485)
0620 02:36:15.852183 (+ 9780us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:15.852222 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:15.852434 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:15.853226 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:15.853243 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:15.855523 (+ 2280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:15.855851 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.855879 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.857884 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.857894 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.860892 (+ 2998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.860924 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.862803 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.862819 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.863301 (+ 482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.864653 (+ 1352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:15.864687 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:15.870346 (+ 5659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:15.880394 (+ 10048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.880399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.880403 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.881032 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.881521 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.881527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.881534 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.881702 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.882429 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:15.882436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:15.882612 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 02:36:15.882931 (+ 319us) client_negotiation.cc:240] Negotiation successful
0620 02:36:15.883603 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9601,"mutex_wait_us":162,"spinlock_wait_cycles":26496,"thread_start_us":10152,"threads_started":1}
I20260620 02:36:15.887040 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.842381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36485 (local address 127.0.97.62:36009)
0620 02:36:15.847941 (+ 5560us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.847947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.852351 (+ 4404us) server_negotiation.cc:407] Connection header received
0620 02:36:15.852591 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.852597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.852703 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.852810 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:15.856072 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.857625 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.861173 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.862104 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.862271 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.873726 (+ 11455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:15.873737 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:15.873750 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:15.873802 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:15.880239 (+ 6437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.881211 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.881218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.881222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.881382 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.881847 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.881851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.881855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.882232 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:15.882370 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.886358 (+ 3988us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.886627 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5364,"thread_start_us":5250,"threads_started":1}
I20260620 02:36:15.888070 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.842060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44071 (local address 127.0.97.3:43929)
0620 02:36:15.843212 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:15.843291 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:15.846915 (+ 3624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:15.854811 (+ 7896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:15.854815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:15.855275 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:15.855820 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.855880 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.857681 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.857687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.860898 (+ 3211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.860924 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.862851 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.862862 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.863730 (+ 868us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.864520 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:15.864629 (+ 109us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:15.870998 (+ 6369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:15.878998 (+ 8000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.879011 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.879037 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.879771 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.880896 (+ 1125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.880900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.880904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.881214 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.881947 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:15.881957 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:15.882585 (+ 628us) client_negotiation.cc:769] Sending connection context
0620 02:36:15.882930 (+ 345us) client_negotiation.cc:240] Negotiation successful
0620 02:36:15.887600 (+ 4670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":4510,"thread_start_us":443,"threads_started":1}
I20260620 02:36:15.889559 7811 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36009
I20260620 02:36:15.889678 7812 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44071
I20260620 02:36:15.889953 7812 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:15.889959 7811 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:15.890260 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.841397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49227 (local address 127.0.97.60:42509)
0620 02:36:15.841916 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.841923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.853286 (+ 11363us) server_negotiation.cc:407] Connection header received
0620 02:36:15.853542 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.853550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.853651 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.853744 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:15.856373 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.857504 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.870343 (+ 12839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.871374 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.871562 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.882346 (+ 10784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:15.882353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:15.882358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:15.882402 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:15.886967 (+ 4565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.887820 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.887849 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.887854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.888013 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.888494 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.888499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.888502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.888880 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:15.889019 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.889570 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.889867 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":1237,"threads_started":1}
I20260620 02:36:15.891031 7812 heartbeater.cc:511] Master 127.0.97.61:44071 requested a full tablet report, sending...
I20260620 02:36:15.891314 7811 heartbeater.cc:511] Master 127.0.97.62:36009 requested a full tablet report, sending...
I20260620 02:36:15.892817 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.842060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42509 (local address 127.0.97.3:49227)
0620 02:36:15.853180 (+ 11120us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:15.853198 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:15.853377 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:15.853993 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:15.853997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:15.854691 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:15.856196 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.856207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.857685 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.857689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.870188 (+ 12499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.870212 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.873095 (+ 2883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.873104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.873395 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.876647 (+ 3252us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:15.876677 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:15.882105 (+ 5428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:15.887110 (+ 5005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.887116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.887121 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.887657 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.888180 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:15.888184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:15.888187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:15.888357 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:15.889104 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:15.889108 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:15.889286 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 02:36:15.890524 (+ 1238us) client_negotiation.cc:240] Negotiation successful
0620 02:36:15.890878 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10952,"mutex_wait_us":2226,"thread_start_us":8255,"threads_started":1}
I20260620 02:36:15.893189 7156 ts_manager.cc:194] Registered new tserver with Master: 056f42312e944df685297ea3645a0ede (127.0.97.3:34605)
I20260620 02:36:15.893792 7810 heartbeater.cc:347] Connected to a master server at 127.0.97.60:42509
I20260620 02:36:15.893863 6556 ts_manager.cc:194] Re-registered known tserver with Master: 056f42312e944df685297ea3645a0ede (127.0.97.3:34605)
I20260620 02:36:15.893929 7810 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:15.894397 7810 heartbeater.cc:511] Master 127.0.97.60:42509 requested a full tablet report, sending...
I20260620 02:36:15.895138 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36485
I20260620 02:36:15.895726 7344 ts_manager.cc:194] Registered new tserver with Master: 056f42312e944df685297ea3645a0ede (127.0.97.3:34605)
I20260620 02:36:15.901023 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.867040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.0.97.61:44071)
0620 02:36:15.867654 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.867660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.869986 (+ 2326us) server_negotiation.cc:407] Connection header received
0620 02:36:15.870061 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.870066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.870147 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.870250 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:15.876763 (+ 6513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.878123 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.882648 (+ 4525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.883328 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.883425 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.887374 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:15.887382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:15.887388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:15.887434 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:15.893450 (+ 6016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.897746 (+ 4296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.897753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.897757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.897928 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.898616 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.898621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.898624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.899001 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:15.899271 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.900286 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.900540 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":341,"threads_started":1}
I20260620 02:36:15.904924 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.860586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49182 (local address 127.0.97.62:36009)
0620 02:36:15.875246 (+ 14660us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.875254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.875305 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:36:15.875374 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.875381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.875495 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.875614 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:15.878349 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.880026 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.888472 (+ 8446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.889173 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.889318 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.897463 (+ 8145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:15.897470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:15.897475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:15.897518 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:15.901750 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.902753 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.902760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.902764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.902908 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.903310 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.903316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.903319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.903606 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:15.903931 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.904297 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.904525 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14478,"thread_start_us":15160,"threads_started":1}
I20260620 02:36:15.908849 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.867108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.0.97.60:42509)
0620 02:36:15.868099 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.868107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.876063 (+ 7956us) server_negotiation.cc:407] Connection header received
0620 02:36:15.876138 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.876143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.876224 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.876334 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:15.885032 (+ 8698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.886354 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.895797 (+ 9443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.896583 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.896746 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.901300 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:15.901309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:15.901314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:15.901359 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:15.904512 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.905344 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.905352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.905359 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.905504 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:15.905978 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:15.905983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:15.905985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:15.906994 (+ 1009us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:15.907278 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.908146 (+ 868us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.908428 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":778,"threads_started":1}
I20260620 02:36:15.909973 6556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:36:15.918812 6556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:36:15.958071 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.941386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.0.97.3:34605)
0620 02:36:15.942272 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.942281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.942314 (+ 33us) server_negotiation.cc:407] Connection header received
0620 02:36:15.942417 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.942425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.944032 (+ 1607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.944314 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:15.945031 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.947337 (+ 2306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.954495 (+ 7158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.956834 (+ 2339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.957034 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.957286 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.957577 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.957758 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":458,"threads_started":1}
I20260620 02:36:15.958107 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.940970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34605 (local address 127.0.0.1:39504)
0620 02:36:15.941768 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:15.941790 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:15.941931 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:15.944306 (+ 2375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:15.944316 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:15.944356 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:15.944797 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.944814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.948777 (+ 3963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.948786 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.951808 (+ 3022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.952047 (+ 239us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.957073 (+ 5026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.957080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.957309 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.957393 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 02:36:15.957518 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 02:36:15.957685 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":29,"thread_start_us":487,"threads_started":1}
I20260620 02:36:15.963142 7745 tablet_service.cc:1467] Processing CreateTablet for tablet 147bc596b2ae4839abac3798ea874ba4 (DEFAULT_TABLE table=test-table [id=c94fa220fcc74659902a5025d5ccd6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:15.964133 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.942842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.0.97.1:42313)
0620 02:36:15.955615 (+ 12773us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.955629 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.955685 (+ 56us) server_negotiation.cc:407] Connection header received
0620 02:36:15.956571 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.956594 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.956910 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.957199 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:15.957813 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.959413 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.961514 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.962924 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.963102 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.963328 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.963646 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.963802 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12503,"thread_start_us":13375,"threads_started":1}
I20260620 02:36:15.964185 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.942367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.0.1:54784)
0620 02:36:15.953903 (+ 11536us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:15.953934 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:15.954096 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:15.957134 (+ 3038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:15.957149 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:15.957177 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:15.957610 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.957624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.959530 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.959536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.961361 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.961374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.963185 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.963192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.963433 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.963458 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:36:15.963569 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 02:36:15.963733 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":29,"thread_start_us":419,"threads_started":1}
I20260620 02:36:15.964888 7745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147bc596b2ae4839abac3798ea874ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:15.968297 7458 tablet_service.cc:1467] Processing CreateTablet for tablet 147bc596b2ae4839abac3798ea874ba4 (DEFAULT_TABLE table=test-table [id=c94fa220fcc74659902a5025d5ccd6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:15.969657 7458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147bc596b2ae4839abac3798ea874ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:15.973068 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.943569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.0.97.2:41019)
0620 02:36:15.956815 (+ 13246us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:15.956823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:15.956849 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:36:15.956937 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:15.956946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:15.957930 (+ 984us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:15.958148 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:15.958932 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.962458 (+ 3526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.969885 (+ 7427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.971777 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.972008 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.972238 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:15.972555 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 02:36:15.972758 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13104,"thread_start_us":13025,"threads_started":1}
I20260620 02:36:15.973127 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:15.943153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41019 (local address 127.0.0.1:34984)
0620 02:36:15.956314 (+ 13161us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:15.956334 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:15.956461 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:15.958382 (+ 1921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:15.958388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:15.958416 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:15.958746 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.958756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.962616 (+ 3860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.962622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.965260 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:15.965277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:15.972102 (+ 6825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:15.972110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:15.972329 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:15.972377 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 02:36:15.972492 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 02:36:15.972662 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13037,"mutex_wait_us":30,"thread_start_us":12796,"threads_started":1}
I20260620 02:36:15.976996 7603 tablet_service.cc:1467] Processing CreateTablet for tablet 147bc596b2ae4839abac3798ea874ba4 (DEFAULT_TABLE table=test-table [id=c94fa220fcc74659902a5025d5ccd6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:15.978771 7603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147bc596b2ae4839abac3798ea874ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:15.982035 7846 tablet_bootstrap.cc:492] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: Bootstrap starting.
I20260620 02:36:15.986815 7846 tablet_bootstrap.cc:654] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:15.987119 7847 tablet_bootstrap.cc:492] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: Bootstrap starting.
I20260620 02:36:15.988904 7846 log.cc:824] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:15.991025 7847 tablet_bootstrap.cc:654] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:15.992304 7847 log.cc:824] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:15.992508 7846 tablet_bootstrap.cc:492] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: No bootstrap required, opened a new log
I20260620 02:36:15.992800 7846 ts_tablet_manager.cc:1397] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 02:36:15.999087 7850 tablet_bootstrap.cc:492] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: Bootstrap starting.
I20260620 02:36:16.004511 7850 tablet_bootstrap.cc:654] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:16.004179 7846 raft_consensus.cc:348] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } } peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } }
I20260620 02:36:16.004706 7846 raft_consensus.cc:374] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:16.004815 7846 raft_consensus.cc:729] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056f42312e944df685297ea3645a0ede, State: Initialized, Role: FOLLOWER
I20260620 02:36:16.005548 7846 consensus_queue.cc:260] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } } peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } }
I20260620 02:36:16.005805 7850 log.cc:824] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:16.007207 7847 tablet_bootstrap.cc:492] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: No bootstrap required, opened a new log
I20260620 02:36:16.007530 7847 ts_tablet_manager.cc:1397] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.016s
I20260620 02:36:16.007635 7846 ts_tablet_manager.cc:1428] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 02:36:16.007985 7811 heartbeater.cc:503] Master 127.0.97.62:36009 was elected leader, sending a full tablet report...
I20260620 02:36:16.018566 7850 tablet_bootstrap.cc:492] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: No bootstrap required, opened a new log
I20260620 02:36:16.018952 7850 ts_tablet_manager.cc:1397] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20260620 02:36:16.018886 7847 raft_consensus.cc:348] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } } peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } }
I20260620 02:36:16.019435 7847 raft_consensus.cc:374] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:16.019549 7847 raft_consensus.cc:729] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62ac2bd7747422ebb0a58bcb681f6e1, State: Initialized, Role: FOLLOWER
I20260620 02:36:16.020277 7847 consensus_queue.cc:260] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } } peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } }
I20260620 02:36:16.022186 7847 ts_tablet_manager.cc:1428] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260620 02:36:16.030607 7850 raft_consensus.cc:348] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } } peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } }
I20260620 02:36:16.031123 7850 raft_consensus.cc:374] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:16.031214 7850 raft_consensus.cc:729] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94cd1db181e54aeca840315161cb9e67, State: Initialized, Role: FOLLOWER
W20260620 02:36:16.031621 7528 tablet.cc:2369] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:36:16.031898 7850 consensus_queue.cc:260] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } } peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } }
I20260620 02:36:16.033811 7850 ts_tablet_manager.cc:1428] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260620 02:36:16.034945 7669 heartbeater.cc:503] Master 127.0.97.62:36009 was elected leader, sending a full tablet report...
W20260620 02:36:16.098970 7813 tablet.cc:2369] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:36:16.105475 7854 raft_consensus.cc:482] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:36:16.105876 7854 raft_consensus.cc:504] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } } peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } }
I20260620 02:36:16.108137 7854 leader_election.cc:288] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 056f42312e944df685297ea3645a0ede (127.0.97.3:34605), c62ac2bd7747422ebb0a58bcb681f6e1 (127.0.97.1:42313)
I20260620 02:36:16.117434 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:16.108627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47477 (local address 127.0.97.1:42313)
0620 02:36:16.108948 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:16.108959 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:16.109874 (+ 915us) server_negotiation.cc:407] Connection header received
0620 02:36:16.110237 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:16.110245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:16.110364 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:16.110464 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:16.111932 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.112804 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.115385 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.116316 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.116429 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:16.116516 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:16.117002 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 02:36:16.117146 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 02:36:16.117497 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:16.108864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43681 (local address 127.0.97.3:34605)
0620 02:36:16.109186 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:16.109196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:16.110362 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 02:36:16.110598 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:16.110603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:16.110692 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:16.110829 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:16.112234 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.113383 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.115458 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.116493 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.116661 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:16.116788 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:16.117068 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 02:36:16.117212 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 02:36:16.117857 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:16.108619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34605 (local address 127.0.97.2:43681)
0620 02:36:16.110162 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:16.110185 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:16.110350 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:16.110954 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:16.110959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:16.110993 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:16.111949 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:16.111961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.113590 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.113595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:16.115239 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:16.115252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.116735 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.116741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:16.116859 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:16.116882 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:36:16.117004 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 02:36:16.117146 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1404,"thread_start_us":1005,"threads_started":1}
I20260620 02:36:16.117941 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:16.108479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.97.2:47477)
0620 02:36:16.109714 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:16.109732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:16.109992 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:16.110597 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:16.110604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:16.110639 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:16.111649 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:16.111665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.113146 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.113153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:16.115150 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:16.115170 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.116553 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.116563 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:16.116786 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:16.116836 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 02:36:16.116938 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 02:36:16.117139 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"mutex_wait_us":99,"thread_start_us":735,"threads_started":1}
I20260620 02:36:16.119287 7478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147bc596b2ae4839abac3798ea874ba4" candidate_uuid: "94cd1db181e54aeca840315161cb9e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" is_pre_election: true
I20260620 02:36:16.119222 7765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147bc596b2ae4839abac3798ea874ba4" candidate_uuid: "94cd1db181e54aeca840315161cb9e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056f42312e944df685297ea3645a0ede" is_pre_election: true
I20260620 02:36:16.119900 7478 raft_consensus.cc:2454] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94cd1db181e54aeca840315161cb9e67 in term 0.
I20260620 02:36:16.119900 7765 raft_consensus.cc:2454] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94cd1db181e54aeca840315161cb9e67 in term 0.
I20260620 02:36:16.121022 7556 leader_election.cc:302] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056f42312e944df685297ea3645a0ede, 94cd1db181e54aeca840315161cb9e67; no voters:
I20260620 02:36:16.121685 7854 raft_consensus.cc:2781] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:36:16.121855 7854 raft_consensus.cc:482] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:36:16.121938 7854 raft_consensus.cc:3037] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:16.124873 7854 raft_consensus.cc:504] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } } peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } }
I20260620 02:36:16.125512 7854 leader_election.cc:288] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [CANDIDATE]: Term 1 election: Requested vote from peers 056f42312e944df685297ea3645a0ede (127.0.97.3:34605), c62ac2bd7747422ebb0a58bcb681f6e1 (127.0.97.1:42313)
I20260620 02:36:16.126161 7765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147bc596b2ae4839abac3798ea874ba4" candidate_uuid: "94cd1db181e54aeca840315161cb9e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056f42312e944df685297ea3645a0ede"
I20260620 02:36:16.126256 7478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147bc596b2ae4839abac3798ea874ba4" candidate_uuid: "94cd1db181e54aeca840315161cb9e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1"
I20260620 02:36:16.126421 7765 raft_consensus.cc:3037] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:16.126484 7478 raft_consensus.cc:3037] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:16.128859 7478 raft_consensus.cc:2454] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94cd1db181e54aeca840315161cb9e67 in term 1.
I20260620 02:36:16.128866 7765 raft_consensus.cc:2454] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94cd1db181e54aeca840315161cb9e67 in term 1.
I20260620 02:36:16.129410 7556 leader_election.cc:302] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056f42312e944df685297ea3645a0ede, 94cd1db181e54aeca840315161cb9e67; no voters:
I20260620 02:36:16.129923 7854 raft_consensus.cc:2781] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:16.130723 7854 raft_consensus.cc:686] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 1 LEADER]: Becoming Leader. State: Replica: 94cd1db181e54aeca840315161cb9e67, State: Running, Role: LEADER
I20260620 02:36:16.131702 7854 consensus_queue.cc:237] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } } peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } }
I20260620 02:36:16.137995 6556 catalog_manager.cc:5495] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94cd1db181e54aeca840315161cb9e67 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "94cd1db181e54aeca840315161cb9e67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } health_report { overall_health: HEALTHY } } }
W20260620 02:36:16.192152 7671 tablet.cc:2369] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:36:16.203017 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:16.195818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.0.97.2:41019)
0620 02:36:16.196063 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:16.196068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:16.196094 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:36:16.196341 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:16.196346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:16.196434 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:16.196566 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:36:16.197006 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.197890 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.200174 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.200846 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.200972 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:16.201629 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:36:16.202206 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:36:16.202341 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:16.202577 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 02:36:16.202661 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:36:16.212706 7478 raft_consensus.cc:1261] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1 [term 1 FOLLOWER]: Refusing update from remote peer 94cd1db181e54aeca840315161cb9e67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:16.212668 7765 raft_consensus.cc:1261] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede [term 1 FOLLOWER]: Refusing update from remote peer 94cd1db181e54aeca840315161cb9e67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:16.214272 7854 consensus_queue.cc:1035] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:16.214694 7857 consensus_queue.cc:1035] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:16.229362 7861 mvcc.cc:205] Tried to move back new op lower bound from 7298756510554857472 to 7298756510240038912. Current Snapshot: MvccSnapshot[applied={T|T < 7298756510554857472}]
I20260620 02:36:16.230754 7863 mvcc.cc:205] Tried to move back new op lower bound from 7298756510554857472 to 7298756510240038912. Current Snapshot: MvccSnapshot[applied={T|T < 7298756510554857472}]
I20260620 02:36:16.241122 7862 mvcc.cc:205] Tried to move back new op lower bound from 7298756510554857472 to 7298756510240038912. Current Snapshot: MvccSnapshot[applied={T|T < 7298756510554857472}]
I20260620 02:36:16.249411 6555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:16.249758 6555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:16.259613 6555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b53dba5fad064c26a173bd4691e8a17c: Sending DeleteTablet for 3 replicas of tablet 147bc596b2ae4839abac3798ea874ba4
I20260620 02:36:16.261089 7745 tablet_service.cc:1514] Processing DeleteTablet for tablet 147bc596b2ae4839abac3798ea874ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:16 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39504
I20260620 02:36:16.261401 7458 tablet_service.cc:1514] Processing DeleteTablet for tablet 147bc596b2ae4839abac3798ea874ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:16 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54784
I20260620 02:36:16.261489 7603 tablet_service.cc:1514] Processing DeleteTablet for tablet 147bc596b2ae4839abac3798ea874ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:16 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34984
I20260620 02:36:16.273778 7871 tablet_replica.cc:330] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: stopping tablet replica
I20260620 02:36:16.274497 7871 raft_consensus.cc:2229] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:16.275166 7871 raft_consensus.cc:2258] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:16.277761 7871 ts_tablet_manager.cc:1905] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:16.286513 7871 ts_tablet_manager.cc:1918] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:16.287474 7871 log.cc:1196] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/147bc596b2ae4839abac3798ea874ba4
I20260620 02:36:16.288439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 6524
I20260620 02:36:16.288830 7871 ts_tablet_manager.cc:1939] T 147bc596b2ae4839abac3798ea874ba4 P c62ac2bd7747422ebb0a58bcb681f6e1: Deleting consensus metadata
I20260620 02:36:16.290103 6586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:16.294220 7872 tablet_replica.cc:330] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: stopping tablet replica
I20260620 02:36:16.294775 7872 raft_consensus.cc:2229] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:36:16.295379 7872 raft_consensus.cc:2258] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:16.295890 7870 tablet_replica.cc:330] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: stopping tablet replica
I20260620 02:36:16.296387 7870 raft_consensus.cc:2229] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:16.297022 7870 raft_consensus.cc:2258] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:16.303483 7872 ts_tablet_manager.cc:1905] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:16.307246 7870 ts_tablet_manager.cc:1905] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:16.315109 7872 ts_tablet_manager.cc:1918] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:16.315248 7870 ts_tablet_manager.cc:1918] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:16.315486 7870 log.cc:1196] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/147bc596b2ae4839abac3798ea874ba4
I20260620 02:36:16.315528 7872 log.cc:1196] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/147bc596b2ae4839abac3798ea874ba4
I20260620 02:36:16.316134 7870 ts_tablet_manager.cc:1939] T 147bc596b2ae4839abac3798ea874ba4 P 056f42312e944df685297ea3645a0ede: Deleting consensus metadata
I20260620 02:36:16.316771 7872 ts_tablet_manager.cc:1939] T 147bc596b2ae4839abac3798ea874ba4 P 94cd1db181e54aeca840315161cb9e67: Deleting consensus metadata
I20260620 02:36:16.335695 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:16.264107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.0.97.61:44071)
0620 02:36:16.264306 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:16.264313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:16.286721 (+ 22408us) server_negotiation.cc:407] Connection header received
0620 02:36:16.287017 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:16.287023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:16.287141 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:16.287289 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:16.300854 (+ 13565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.302014 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.305635 (+ 3621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.306356 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.306531 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:16.327986 (+ 21455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:16.327995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:16.328000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:16.328057 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:16.332383 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:16.333422 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:16.333428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:16.333433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:16.333610 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:16.334046 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:16.334052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:16.334056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:16.334455 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:16.334703 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:16.335094 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 02:36:16.335368 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:36:16.354542 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:16.287590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.0.97.60:42509)
0620 02:36:16.287806 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:16.287813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:16.335933 (+ 48120us) server_negotiation.cc:407] Connection header received
0620 02:36:16.336177 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:16.336184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:16.336264 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:16.336331 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:16.338265 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.339446 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.341726 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.342408 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.342587 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:16.346444 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:16.346455 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:16.346461 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:16.346515 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:16.351025 (+ 4510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:16.352122 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:16.352130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:16.352134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:16.352295 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:16.352726 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:16.352732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:16.352736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:16.353220 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:16.353493 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:16.353859 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 02:36:16.354178 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:36:16.430352 6540 catalog_manager.cc:4841] TS 056f42312e944df685297ea3645a0ede (127.0.97.3:34605): tablet 147bc596b2ae4839abac3798ea874ba4 (table test-table [id=c94fa220fcc74659902a5025d5ccd6f3]) successfully deleted
I20260620 02:36:16.431403 6542 catalog_manager.cc:4841] TS 94cd1db181e54aeca840315161cb9e67 (127.0.97.2:41019): tablet 147bc596b2ae4839abac3798ea874ba4 (table test-table [id=c94fa220fcc74659902a5025d5ccd6f3]) successfully deleted
I20260620 02:36:16.431692 6542 catalog_manager.cc:4841] TS c62ac2bd7747422ebb0a58bcb681f6e1 (127.0.97.1:42313): tablet 147bc596b2ae4839abac3798ea874ba4 (table test-table [id=c94fa220fcc74659902a5025d5ccd6f3]) successfully deleted
I20260620 02:36:16.432310 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 6524
I20260620 02:36:16.482448 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:16.463091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.0.97.60:42509)
0620 02:36:16.463280 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:16.463286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:16.464624 (+ 1338us) server_negotiation.cc:407] Connection header received
0620 02:36:16.464976 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:16.464984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:16.465084 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:16.465176 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:16.468260 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.469021 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.470934 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.471521 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.471608 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:16.475770 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:16.475776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:16.475779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:16.475818 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:16.478909 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:16.480021 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:16.480028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:16.480031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:16.480191 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:16.480686 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:16.480693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:16.480696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:16.480994 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:16.481161 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:16.481936 (+ 775us) server_negotiation.cc:299] Negotiation successful
0620 02:36:16.482145 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:36:16.482797 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:16.463360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.0.97.61:44071)
0620 02:36:16.463633 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:16.463642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:16.465388 (+ 1746us) server_negotiation.cc:407] Connection header received
0620 02:36:16.465456 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:16.465461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:16.465540 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:16.465640 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:16.468316 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.469443 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.471294 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:16.471928 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:16.472046 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:16.475719 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:16.475727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:16.475732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:16.475784 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:16.479374 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:16.480186 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:16.480192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:16.480196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:16.480343 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:16.480757 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:16.480764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:16.480768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:16.481128 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:16.481362 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:16.481939 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 02:36:16.482221 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:36:17.239182 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.238423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.2:50959)
0620 02:36:17.238899 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.239018 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":256,"threads_started":1}
W20260620 02:36:17.239745 7669 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
I20260620 02:36:17.242761 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.241872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.1:44471)
0620 02:36:17.242353 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.242526 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":251,"threads_started":1}
W20260620 02:36:17.243273 7524 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
I20260620 02:36:17.251812 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.250497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.3:51945)
0620 02:36:17.251323 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.251532 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":536,"threads_started":1}
W20260620 02:36:17.252244 7811 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36009 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
I20260620 02:36:17.758889 7884 raft_consensus.cc:482] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b53dba5fad064c26a173bd4691e8a17c)
I20260620 02:36:17.759222 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b53dba5fad064c26a173bd4691e8a17c)
I20260620 02:36:17.759233 7884 raft_consensus.cc:504] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } }
I20260620 02:36:17.759418 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } }
I20260620 02:36:17.760789 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009), f48ad3e294f44c9ba44e8baa08401195 (127.0.97.61:44071)
I20260620 02:36:17.760855 7884 leader_election.cc:288] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009), 809c672b0cdc4956ada9756e748673bc (127.0.97.60:42509)
I20260620 02:36:17.776715 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.764454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49608)
0620 02:36:17.776032 (+ 11578us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.776339 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11355,"thread_start_us":11447,"threads_started":1}
W20260620 02:36:17.777681 7329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
I20260620 02:36:17.778931 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.760991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49614)
0620 02:36:17.778072 (+ 17081us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.778571 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16932,"thread_start_us":17307,"threads_started":1}
W20260620 02:36:17.779179 7142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
I20260620 02:36:17.780364 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.779766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49634)
0620 02:36:17.780019 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.780152 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260620 02:36:17.780682 7329 leader_election.cc:334] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009): Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
I20260620 02:36:17.786226 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.785691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49636)
0620 02:36:17.785921 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.786037 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 02:36:17.786476 7142 leader_election.cc:334] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009): Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
I20260620 02:36:17.790199 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.761349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42509 (local address 127.0.0.1:53208)
0620 02:36:17.778589 (+ 17240us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.778608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:17.778786 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:17.779429 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:17.779436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:17.779474 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:17.783855 (+ 4381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:17.783888 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:17.785627 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:17.785633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:17.787941 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:17.787956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:17.789501 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:17.789506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:17.789685 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:17.789712 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 02:36:17.789782 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 02:36:17.789884 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17102,"mutex_wait_us":70,"thread_start_us":16971,"threads_started":1}
I20260620 02:36:17.790403 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.761534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.0.97.60:42509)
0620 02:36:17.773942 (+ 12408us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:17.773949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:17.778842 (+ 4893us) server_negotiation.cc:407] Connection header received
0620 02:36:17.778932 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:17.778938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:17.779023 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:17.779141 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:17.784237 (+ 5096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:17.785449 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:17.788138 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:17.789324 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:17.789463 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:17.789567 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:17.789871 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 02:36:17.790018 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":12239,"thread_start_us":12026,"threads_started":1}
I20260620 02:36:17.791155 7354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f48ad3e294f44c9ba44e8baa08401195" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "809c672b0cdc4956ada9756e748673bc" is_pre_election: true
I20260620 02:36:17.791538 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.761015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44071 (local address 127.0.0.1:35248)
0620 02:36:17.774267 (+ 13252us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.774299 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:17.774512 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:17.784874 (+ 10362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:17.784878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:17.784907 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:17.785249 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:17.785259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:17.786995 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:17.787001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:17.788838 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:17.788851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:17.790655 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:17.790661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:17.790868 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:17.790901 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:36:17.790987 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 02:36:17.791089 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13042,"mutex_wait_us":72,"thread_start_us":12818,"threads_started":1}
I20260620 02:36:17.791553 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.778653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.0.97.61:44071)
0620 02:36:17.784528 (+ 5875us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:17.784534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:17.784552 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:36:17.784611 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:17.784616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:17.784695 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:17.784810 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:17.785406 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:17.786852 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:17.789010 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:17.790287 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:17.790409 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:17.790528 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:17.791048 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 02:36:17.791191 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":5744,"thread_start_us":5575,"threads_started":1}
I20260620 02:36:17.791615 7354 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f48ad3e294f44c9ba44e8baa08401195 in term 1.
I20260620 02:36:17.792183 7165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "809c672b0cdc4956ada9756e748673bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f48ad3e294f44c9ba44e8baa08401195" is_pre_election: true
I20260620 02:36:17.792420 7165 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809c672b0cdc4956ada9756e748673bc in term 1.
I20260620 02:36:17.792824 7330 leader_election.cc:302] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 809c672b0cdc4956ada9756e748673bc, f48ad3e294f44c9ba44e8baa08401195; no voters: b53dba5fad064c26a173bd4691e8a17c
I20260620 02:36:17.792835 7140 leader_election.cc:302] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 809c672b0cdc4956ada9756e748673bc, f48ad3e294f44c9ba44e8baa08401195; no voters: b53dba5fad064c26a173bd4691e8a17c
I20260620 02:36:17.793156 7885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 02:36:17.793232 7884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 02:36:17.793303 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 FOLLOWER]: Starting leader election (detected failure of leader b53dba5fad064c26a173bd4691e8a17c)
I20260620 02:36:17.793357 7884 raft_consensus.cc:482] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b53dba5fad064c26a173bd4691e8a17c)
I20260620 02:36:17.793402 7885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:36:17.793458 7884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:36:17.795688 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } }
I20260620 02:36:17.795794 7884 raft_consensus.cc:504] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } }
I20260620 02:36:17.796505 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 2 election: Requested vote from peers b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009), f48ad3e294f44c9ba44e8baa08401195 (127.0.97.61:44071)
I20260620 02:36:17.796583 7884 leader_election.cc:288] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [CANDIDATE]: Term 2 election: Requested vote from peers b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009), 809c672b0cdc4956ada9756e748673bc (127.0.97.60:42509)
I20260620 02:36:17.797112 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.796672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49648)
0620 02:36:17.796857 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.796944 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 02:36:17.797293 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.796868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49656)
0620 02:36:17.797038 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.797147 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 02:36:17.797341 7354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f48ad3e294f44c9ba44e8baa08401195" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "809c672b0cdc4956ada9756e748673bc"
I20260620 02:36:17.797632 7354 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f48ad3e294f44c9ba44e8baa08401195 in current term 2: Already voted for candidate 809c672b0cdc4956ada9756e748673bc in this term.
I20260620 02:36:17.798228 7165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "809c672b0cdc4956ada9756e748673bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f48ad3e294f44c9ba44e8baa08401195"
I20260620 02:36:17.798368 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.797932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49662)
0620 02:36:17.798132 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.798228 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 02:36:17.798393 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:17.797937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49672)
0620 02:36:17.798151 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:17.798231 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 02:36:17.798513 7165 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 809c672b0cdc4956ada9756e748673bc in current term 2: Already voted for candidate f48ad3e294f44c9ba44e8baa08401195 in this term.
W20260620 02:36:17.798547 7329 leader_election.cc:334] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009): Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
W20260620 02:36:17.798601 7142 leader_election.cc:334] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009): Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
I20260620 02:36:17.798702 7142 leader_election.cc:302] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f48ad3e294f44c9ba44e8baa08401195; no voters: 809c672b0cdc4956ada9756e748673bc, b53dba5fad064c26a173bd4691e8a17c
I20260620 02:36:17.798979 7884 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 02:36:17.798979 7330 leader_election.cc:302] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 809c672b0cdc4956ada9756e748673bc; no voters: b53dba5fad064c26a173bd4691e8a17c, f48ad3e294f44c9ba44e8baa08401195
I20260620 02:36:17.799232 7885 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 02:36:18.241298 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:18.240505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.2:40725)
0620 02:36:18.240974 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:18.241059 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":247,"threads_started":1}
I20260620 02:36:18.244860 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:18.244064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.1:39661)
0620 02:36:18.244565 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:18.244696 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":314,"threads_started":1}
I20260620 02:36:18.254321 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:18.252938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.3:52269)
0620 02:36:18.253880 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:18.254083 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":741,"threads_started":1}
I20260620 02:36:19.243571 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.242514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.2:34121)
0620 02:36:19.243231 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.243330 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":320,"threads_started":1}
I20260620 02:36:19.246788 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.246004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.1:42377)
0620 02:36:19.246501 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.246593 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":281,"threads_started":1}
I20260620 02:36:19.256556 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.255249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.3:42257)
0620 02:36:19.256265 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.256353 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":791,"threads_started":1}
I20260620 02:36:19.676005 7900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:36:19.676290 7900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } }
I20260620 02:36:19.677026 7900 leader_election.cc:288] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009), f48ad3e294f44c9ba44e8baa08401195 (127.0.97.61:44071)
I20260620 02:36:19.677644 7165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "809c672b0cdc4956ada9756e748673bc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f48ad3e294f44c9ba44e8baa08401195" is_pre_election: true
I20260620 02:36:19.677896 7165 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809c672b0cdc4956ada9756e748673bc in term 2.
I20260620 02:36:19.678201 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.677317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49936)
0620 02:36:19.677865 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.677973 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":272,"threads_started":1}
I20260620 02:36:19.678386 7330 leader_election.cc:302] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809c672b0cdc4956ada9756e748673bc, f48ad3e294f44c9ba44e8baa08401195; no voters:
I20260620 02:36:19.678721 7900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 02:36:19.678894 7900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:36:19.679039 7900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:36:19.679966 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.679526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49944)
0620 02:36:19.679668 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.679791 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:36:19.680150 7329 leader_election.cc:334] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009): Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
I20260620 02:36:19.681550 7900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } }
I20260620 02:36:19.682111 7900 leader_election.cc:288] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 3 election: Requested vote from peers b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009), f48ad3e294f44c9ba44e8baa08401195 (127.0.97.61:44071)
I20260620 02:36:19.682698 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.682291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49950)
0620 02:36:19.682436 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.682528 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 02:36:19.682765 7165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "809c672b0cdc4956ada9756e748673bc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f48ad3e294f44c9ba44e8baa08401195"
I20260620 02:36:19.683038 7165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:36:19.683754 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.683395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49952)
0620 02:36:19.683568 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.683656 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 02:36:19.683933 7329 leader_election.cc:334] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009): Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
I20260620 02:36:19.685424 7165 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809c672b0cdc4956ada9756e748673bc in term 3.
I20260620 02:36:19.685840 7330 leader_election.cc:302] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 809c672b0cdc4956ada9756e748673bc, f48ad3e294f44c9ba44e8baa08401195; no voters: b53dba5fad064c26a173bd4691e8a17c
I20260620 02:36:19.686102 7900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 3 FOLLOWER]: Leader election won for term 3
I20260620 02:36:19.686487 7900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [term 3 LEADER]: Becoming Leader. State: Replica: 809c672b0cdc4956ada9756e748673bc, State: Running, Role: LEADER
I20260620 02:36:19.686839 7900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } }
I20260620 02:36:19.689352 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 809c672b0cdc4956ada9756e748673bc. Latest consensus state: current_term: 3 leader_uuid: "809c672b0cdc4956ada9756e748673bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:19.689805 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:19.690521 7905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:19.694098 7905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:19.695008 7905 catalog_manager.cc:1259] Loaded cluster ID: 909a3d673ad54206ac43ca677703694b
I20260620 02:36:19.695078 7905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:19.696386 7905 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:19.697170 7905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: Loaded TSK: 0
I20260620 02:36:19.698048 7905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:36:19.777010 7344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:36:19.784332 7344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:36:19.788479 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.788034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49978)
0620 02:36:19.788259 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.788336 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 02:36:19.788576 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [term 3 FOLLOWER]: Refusing update from remote peer 809c672b0cdc4956ada9756e748673bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 02:36:19.789314 7903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 02:36:19.789475 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.789113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:49992)
0620 02:36:19.789308 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.789371 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:36:19.789688 7329 consensus_peers.cc:487] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc -> Peer b53dba5fad064c26a173bd4691e8a17c (127.0.97.62:36009): Couldn't send request to peer b53dba5fad064c26a173bd4691e8a17c. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 02:36:19.792459 7907 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 809c672b0cdc4956ada9756e748673bc. Latest consensus state: current_term: 3 leader_uuid: "809c672b0cdc4956ada9756e748673bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:19.792722 7907 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:36:19.794075 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "809c672b0cdc4956ada9756e748673bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:19.794402 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:19.794852 7907 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "809c672b0cdc4956ada9756e748673bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b53dba5fad064c26a173bd4691e8a17c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36009 } } peers { permanent_uuid: "f48ad3e294f44c9ba44e8baa08401195" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44071 } attrs { promote: false } } peers { permanent_uuid: "809c672b0cdc4956ada9756e748673bc" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42509 } attrs { promote: false } } }
I20260620 02:36:19.795107 7907 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48ad3e294f44c9ba44e8baa08401195 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:36:19.813037 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.803684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.0.97.1:42313)
0620 02:36:19.804931 (+ 1247us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:19.804940 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:19.804965 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:36:19.805049 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:19.805057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:19.805153 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:19.805277 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:19.805903 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.807203 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.810496 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.812109 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.812277 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.812432 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:19.812700 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 02:36:19.812843 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095,"thread_start_us":1642,"threads_started":1}
I20260620 02:36:19.813117 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.803547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.0.1:54800)
0620 02:36:19.803731 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.803753 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:19.803898 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:19.805309 (+ 1411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:19.805317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:19.805350 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:19.805737 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:19.805751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.807339 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.807346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:19.810276 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:19.810300 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.812315 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.812321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:19.812514 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.812552 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:36:19.812649 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 02:36:19.812816 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 02:36:19.813833 7458 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4f6c6840704ec5aa598e017e890abc (DEFAULT_TABLE table=test-table [id=57a8b0a96cf2421687d932bfef0fa692]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:19.814316 7458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4f6c6840704ec5aa598e017e890abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:19.819111 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.804307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.0.97.2:41019)
0620 02:36:19.812911 (+ 8604us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:19.812917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:19.812935 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:36:19.813039 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:19.813044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:19.813126 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:19.813950 (+ 824us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:19.814110 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.814821 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.817011 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.818080 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.818220 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.818319 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:19.818639 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 02:36:19.818793 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8498,"thread_start_us":8405,"threads_started":1}
I20260620 02:36:19.819159 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.803935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41019 (local address 127.0.0.1:35002)
0620 02:36:19.804782 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.804806 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:19.804972 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:19.813297 (+ 8325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:19.813304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:19.813332 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:19.813738 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:19.813751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.814986 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.814991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:19.816847 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:19.816859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.818275 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.818282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:19.818458 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.818494 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:36:19.818592 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 02:36:19.818749 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":24,"spinlock_wait_cycles":4352,"thread_start_us":303,"threads_started":1}
I20260620 02:36:19.820024 7603 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4f6c6840704ec5aa598e017e890abc (DEFAULT_TABLE table=test-table [id=57a8b0a96cf2421687d932bfef0fa692]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:19.820184 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.804674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.0.97.3:34605)
0620 02:36:19.812207 (+ 7533us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:19.812216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:19.812236 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:36:19.812309 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:19.812315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:19.812389 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:19.812487 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:19.814081 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.815093 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.817760 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.818949 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.819078 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.819197 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:19.819730 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 02:36:19.819901 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7412,"thread_start_us":7328,"threads_started":1}
I20260620 02:36:19.820469 7603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4f6c6840704ec5aa598e017e890abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:19.820585 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.804349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34605 (local address 127.0.0.1:39506)
0620 02:36:19.811308 (+ 6959us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.811328 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:19.811436 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:19.813413 (+ 1977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:19.813418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:19.813443 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:19.813883 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:19.813894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.815258 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.815264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:19.817500 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:19.817514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.819348 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.819354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:19.819533 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.819579 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 02:36:19.819702 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 02:36:19.819834 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6836,"mutex_wait_us":51,"thread_start_us":7114,"threads_started":1}
I20260620 02:36:19.821404 7745 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4f6c6840704ec5aa598e017e890abc (DEFAULT_TABLE table=test-table [id=57a8b0a96cf2421687d932bfef0fa692]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:19.821705 7918 tablet_bootstrap.cc:492] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1: Bootstrap starting.
I20260620 02:36:19.821831 7745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4f6c6840704ec5aa598e017e890abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:19.823513 7918 tablet_bootstrap.cc:654] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:19.826233 7920 tablet_bootstrap.cc:492] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67: Bootstrap starting.
I20260620 02:36:19.827632 7920 tablet_bootstrap.cc:654] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:19.829277 7918 tablet_bootstrap.cc:492] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1: No bootstrap required, opened a new log
I20260620 02:36:19.829460 7918 ts_tablet_manager.cc:1397] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 02:36:19.829814 7920 tablet_bootstrap.cc:492] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67: No bootstrap required, opened a new log
I20260620 02:36:19.829958 7920 ts_tablet_manager.cc:1397] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:36:19.830618 7920 raft_consensus.cc:348] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } } peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } }
I20260620 02:36:19.830550 7918 raft_consensus.cc:348] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } } peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } }
I20260620 02:36:19.830858 7918 raft_consensus.cc:374] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:19.830977 7918 raft_consensus.cc:729] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62ac2bd7747422ebb0a58bcb681f6e1, State: Initialized, Role: FOLLOWER
I20260620 02:36:19.831549 7920 raft_consensus.cc:374] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:19.831359 7918 consensus_queue.cc:260] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } } peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } }
I20260620 02:36:19.831763 7920 raft_consensus.cc:729] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94cd1db181e54aeca840315161cb9e67, State: Initialized, Role: FOLLOWER
I20260620 02:36:19.832103 7920 consensus_queue.cc:260] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } } peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } }
I20260620 02:36:19.837491 7922 tablet_bootstrap.cc:492] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede: Bootstrap starting.
I20260620 02:36:19.838438 7918 ts_tablet_manager.cc:1428] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1: Time spent starting tablet: real 0.009s user 0.001s sys 0.008s
I20260620 02:36:19.840297 7922 tablet_bootstrap.cc:654] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:19.841395 7523 heartbeater.cc:503] Master 127.0.97.60:42509 was elected leader, sending a full tablet report...
I20260620 02:36:19.851121 7920 ts_tablet_manager.cc:1428] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20260620 02:36:19.851590 7668 heartbeater.cc:503] Master 127.0.97.60:42509 was elected leader, sending a full tablet report...
I20260620 02:36:19.860246 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.833225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.2:54973)
0620 02:36:19.859517 (+ 26292us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.859703 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26114,"thread_start_us":25949,"threads_started":1}
I20260620 02:36:19.861578 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.860948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.2:46965)
0620 02:36:19.861197 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.861391 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 02:36:19.862241 7922 tablet_bootstrap.cc:492] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede: No bootstrap required, opened a new log
I20260620 02:36:19.862397 7922 ts_tablet_manager.cc:1397] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.007s
I20260620 02:36:19.862887 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.844653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.1:45935)
0620 02:36:19.862572 (+ 17919us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.862690 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17752,"threads_started":1}
I20260620 02:36:19.863236 7922 raft_consensus.cc:348] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } } peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } }
I20260620 02:36:19.863482 7922 raft_consensus.cc:374] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:19.863586 7922 raft_consensus.cc:729] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056f42312e944df685297ea3645a0ede, State: Initialized, Role: FOLLOWER
I20260620 02:36:19.863901 7922 consensus_queue.cc:260] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } } peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } }
I20260620 02:36:19.864187 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.863736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.1:37067)
0620 02:36:19.863930 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.864034 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 02:36:19.864889 7922 ts_tablet_manager.cc:1428] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 02:36:19.866323 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.865203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.3:59517)
0620 02:36:19.865951 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.866060 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"threads_started":1}
I20260620 02:36:19.867127 7810 heartbeater.cc:503] Master 127.0.97.60:42509 was elected leader, sending a full tablet report...
I20260620 02:36:19.868896 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.868542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.3:51387)
0620 02:36:19.868696 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.868772 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 02:36:19.881975 7923 raft_consensus.cc:482] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:36:19.882210 7923 raft_consensus.cc:504] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } } peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } }
I20260620 02:36:19.883666 7923 leader_election.cc:288] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94cd1db181e54aeca840315161cb9e67 (127.0.97.2:41019), 056f42312e944df685297ea3645a0ede (127.0.97.3:34605)
I20260620 02:36:19.890080 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.884156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49783 (local address 127.0.97.2:41019)
0620 02:36:19.884388 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:19.884396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:19.884772 (+ 376us) server_negotiation.cc:407] Connection header received
0620 02:36:19.884968 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:19.884974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:19.885038 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:19.885132 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:19.885623 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.886304 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.888268 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.889055 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.889228 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.889339 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:19.889662 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 02:36:19.889791 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:36:19.890170 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.884128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33695 (local address 127.0.97.3:34605)
0620 02:36:19.884385 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:19.884396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:19.884642 (+ 246us) server_negotiation.cc:407] Connection header received
0620 02:36:19.884871 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:19.884875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:19.884932 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:19.885029 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:19.885563 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.886258 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.888380 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.889183 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.889364 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.889478 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:19.889803 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 02:36:19.889933 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 02:36:19.890235 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.883984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34605 (local address 127.0.97.1:33695)
0620 02:36:19.884505 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.884542 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:19.884722 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:19.885042 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:19.885046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:19.885084 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:19.885412 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:19.885424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.886384 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.886392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:19.888222 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:19.888235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.889440 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.889446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:19.889640 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.889663 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:36:19.889778 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 02:36:19.889918 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":202,"spinlock_wait_cycles":1920}
I20260620 02:36:19.890275 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.884055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41019 (local address 127.0.97.1:49783)
0620 02:36:19.884664 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.884678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:19.884835 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:19.885161 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:19.885165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:19.885188 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:19.885481 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:19.885489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.886435 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.886446 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:19.888117 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:19.888131 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.889284 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.889290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:19.889497 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.889523 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 02:36:19.889644 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 02:36:19.889785 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":89,"thread_start_us":287,"threads_started":1}
I20260620 02:36:19.891036 7765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4f6c6840704ec5aa598e017e890abc" candidate_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056f42312e944df685297ea3645a0ede" is_pre_election: true
I20260620 02:36:19.891311 7765 raft_consensus.cc:2454] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c62ac2bd7747422ebb0a58bcb681f6e1 in term 0.
I20260620 02:36:19.891754 7623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4f6c6840704ec5aa598e017e890abc" candidate_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94cd1db181e54aeca840315161cb9e67" is_pre_election: true
I20260620 02:36:19.891947 7411 leader_election.cc:302] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056f42312e944df685297ea3645a0ede, c62ac2bd7747422ebb0a58bcb681f6e1; no voters:
I20260620 02:36:19.892256 7623 raft_consensus.cc:2454] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c62ac2bd7747422ebb0a58bcb681f6e1 in term 0.
I20260620 02:36:19.892386 7923 raft_consensus.cc:2781] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:36:19.892549 7923 raft_consensus.cc:482] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:36:19.892670 7923 raft_consensus.cc:3037] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:19.895092 7923 raft_consensus.cc:504] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } } peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } }
I20260620 02:36:19.896404 7923 leader_election.cc:288] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [CANDIDATE]: Term 1 election: Requested vote from peers 94cd1db181e54aeca840315161cb9e67 (127.0.97.2:41019), 056f42312e944df685297ea3645a0ede (127.0.97.3:34605)
I20260620 02:36:19.896497 7623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4f6c6840704ec5aa598e017e890abc" candidate_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94cd1db181e54aeca840315161cb9e67"
I20260620 02:36:19.896540 7765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4f6c6840704ec5aa598e017e890abc" candidate_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "056f42312e944df685297ea3645a0ede"
I20260620 02:36:19.896749 7765 raft_consensus.cc:3037] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:19.896759 7623 raft_consensus.cc:3037] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:19.899972 7765 raft_consensus.cc:2454] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c62ac2bd7747422ebb0a58bcb681f6e1 in term 1.
I20260620 02:36:19.899972 7623 raft_consensus.cc:2454] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c62ac2bd7747422ebb0a58bcb681f6e1 in term 1.
I20260620 02:36:19.900519 7411 leader_election.cc:302] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056f42312e944df685297ea3645a0ede, c62ac2bd7747422ebb0a58bcb681f6e1; no voters:
I20260620 02:36:19.900874 7923 raft_consensus.cc:2781] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:19.901338 7923 raft_consensus.cc:686] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 1 LEADER]: Becoming Leader. State: Replica: c62ac2bd7747422ebb0a58bcb681f6e1, State: Running, Role: LEADER
I20260620 02:36:19.901804 7923 consensus_queue.cc:237] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } } peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } }
I20260620 02:36:19.902503 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.902021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.1:46725)
0620 02:36:19.902198 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.902350 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 02:36:19.906342 7343 catalog_manager.cc:5495] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c62ac2bd7747422ebb0a58bcb681f6e1 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ac2bd7747422ebb0a58bcb681f6e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 } health_report { overall_health: UNKNOWN } } }
I20260620 02:36:19.928352 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.923047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.0.97.1:42313)
0620 02:36:19.923228 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:19.923234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:19.923294 (+ 60us) server_negotiation.cc:407] Connection header received
0620 02:36:19.923464 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:19.923468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:19.923552 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:19.923680 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:36:19.924045 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.924747 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.926037 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.926480 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.926576 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.927040 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:36:19.927628 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:36:19.927741 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:19.928032 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 02:36:19.928136 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:36:19.932636 7765 raft_consensus.cc:1261] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede [term 1 FOLLOWER]: Refusing update from remote peer c62ac2bd7747422ebb0a58bcb681f6e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:19.933005 7623 raft_consensus.cc:1261] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67 [term 1 FOLLOWER]: Refusing update from remote peer c62ac2bd7747422ebb0a58bcb681f6e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:19.933401 7923 consensus_queue.cc:1035] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "056f42312e944df685297ea3645a0ede" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:19.933775 7931 consensus_queue.cc:1035] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94cd1db181e54aeca840315161cb9e67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:19.936913 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.936394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.2:52317)
0620 02:36:19.936618 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.936733 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 02:36:19.937250 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.936742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.3:42049)
0620 02:36:19.936940 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.937021 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 02:36:19.942842 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.942400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.2:34879)
0620 02:36:19.942546 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.942643 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 02:36:19.949749 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.948883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.1:54211)
0620 02:36:19.949403 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.949505 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367}
I20260620 02:36:19.951025 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.950327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.97.3:42797)
0620 02:36:19.950718 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:19.950859 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260620 02:36:19.961305 7344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:19.961593 7344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:19.967893 7344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 809c672b0cdc4956ada9756e748673bc: Sending DeleteTablet for 3 replicas of tablet 2c4f6c6840704ec5aa598e017e890abc
I20260620 02:36:19.969095 7745 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4f6c6840704ec5aa598e017e890abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:19 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39506
I20260620 02:36:19.969573 7603 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4f6c6840704ec5aa598e017e890abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:19 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:35002
I20260620 02:36:19.970275 7945 tablet_replica.cc:330] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67: stopping tablet replica
I20260620 02:36:19.970280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 7331
I20260620 02:36:19.970530 7945 raft_consensus.cc:2229] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:19.970959 7945 raft_consensus.cc:2258] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:19.971345 7458 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4f6c6840704ec5aa598e017e890abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:19 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:54800
I20260620 02:36:19.972811 7945 ts_tablet_manager.cc:1905] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:19.981134 7945 ts_tablet_manager.cc:1918] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:19.981406 7945 log.cc:1196] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/2c4f6c6840704ec5aa598e017e890abc
I20260620 02:36:19.982033 7945 ts_tablet_manager.cc:1939] T 2c4f6c6840704ec5aa598e017e890abc P 94cd1db181e54aeca840315161cb9e67: Deleting consensus metadata
I20260620 02:36:19.983546 7329 catalog_manager.cc:4841] TS 94cd1db181e54aeca840315161cb9e67 (127.0.97.2:41019): tablet 2c4f6c6840704ec5aa598e017e890abc (table test-table [id=57a8b0a96cf2421687d932bfef0fa692]) successfully deleted
I20260620 02:36:19.988543 7944 tablet_replica.cc:330] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede: stopping tablet replica
I20260620 02:36:19.988778 7944 raft_consensus.cc:2229] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:19.989112 7944 raft_consensus.cc:2258] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:19.989357 7946 tablet_replica.cc:330] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1: stopping tablet replica
I20260620 02:36:19.989583 7946 raft_consensus.cc:2229] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:36:19.990406 7944 ts_tablet_manager.cc:1905] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:19.990593 7946 raft_consensus.cc:2258] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:19.991987 7946 ts_tablet_manager.cc:1905] T 2c4f6c6840704ec5aa598e017e890abc P c62ac2bd7747422ebb0a58bcb681f6e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:19.993717 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:19.972442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.0.97.1:42313)
0620 02:36:19.972651 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:19.972657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:19.972680 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:36:19.972911 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:19.972919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:19.973033 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:19.973109 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:19.975394 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.976826 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.978868 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:19.979718 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:19.979863 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:19.983612 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:19.983627 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:19.983645 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:19.983693 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:19.989756 (+ 6063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:19.990845 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:19.990858 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:19.990862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:19.991080 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:19.991546 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:19.991553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:19.991557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:19.992061 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:19.992407 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:19.993028 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 02:36:19.993282 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:36:19.995769 7518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:20.003105 7944 ts_tablet_manager.cc:1918] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:20.003306 7944 log.cc:1196] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/2c4f6c6840704ec5aa598e017e890abc
I20260620 02:36:20.003700 7944 ts_tablet_manager.cc:1939] T 2c4f6c6840704ec5aa598e017e890abc P 056f42312e944df685297ea3645a0ede: Deleting consensus metadata
I20260620 02:36:20.004971 7327 catalog_manager.cc:4841] TS 056f42312e944df685297ea3645a0ede (127.0.97.3:34605): tablet 2c4f6c6840704ec5aa598e017e890abc (table test-table [id=57a8b0a96cf2421687d932bfef0fa692]) successfully deleted
I20260620 02:36:20.108714 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 7331
I20260620 02:36:20.127638 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 7535
I20260620 02:36:20.128468 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.127793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.0.1:54834)
0620 02:36:20.128113 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:20.128237 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 02:36:20.128791 7329 catalog_manager.cc:4584] TS c62ac2bd7747422ebb0a58bcb681f6e1 (127.0.97.1:42313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c4f6c6840704ec5aa598e017e890abc: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42313: connect: Connection refused (error 111)
I20260620 02:36:20.142081 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.128436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.0.97.2:41019)
0620 02:36:20.128633 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:20.128639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:20.128660 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:36:20.128869 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:20.128873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:20.128975 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:20.129102 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:20.130366 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:20.131100 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:20.132844 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:20.133244 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:20.133387 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:20.136065 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:20.136081 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:20.136097 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:20.136134 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:20.139050 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:20.139821 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:20.139960 (+ 139us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:20.139963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:20.140090 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:20.140513 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:20.140521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:20.140525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:20.140901 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:20.141212 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:20.141506 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 02:36:20.141730 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 02:36:20.142877 7663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:20.182790 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.182149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.0.1:54836)
0620 02:36:20.182493 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:20.182608 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260620 02:36:20.184031 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.183550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.0.1:54838)
0620 02:36:20.183720 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:20.183813 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 02:36:20.203372 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.202863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:50002)
0620 02:36:20.203067 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:20.203168 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 02:36:20.204469 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.204104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36009 (local address 127.0.0.1:50012)
0620 02:36:20.204260 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:20.204347 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 02:36:20.260885 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.260212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.0.1:54840)
0620 02:36:20.260503 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:20.260655 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 02:36:20.261981 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.261650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.0.1:54854)
0620 02:36:20.261791 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:20.261876 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 02:36:20.269577 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 7535
I20260620 02:36:20.293640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 7682
I20260620 02:36:20.310786 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.294491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39516 (local address 127.0.97.3:34605)
0620 02:36:20.294694 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:20.294700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:20.294822 (+ 122us) server_negotiation.cc:407] Connection header received
0620 02:36:20.295079 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:20.295085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:20.295236 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:20.295367 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:20.297410 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:20.298359 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:20.299708 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:20.300191 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:20.300318 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:20.303288 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:20.303303 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:20.303320 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:20.303366 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:20.307324 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:20.308689 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:20.308698 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:20.308701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:20.308869 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:20.309325 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:20.309329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:20.309331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:20.309707 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:20.309947 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:20.310212 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 02:36:20.310395 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:36:20.311503 7805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:20.376925 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.376431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.0.1:54866)
0620 02:36:20.376646 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:20.376747 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 02:36:20.378109 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.377717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.0.1:54870)
0620 02:36:20.377891 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:20.377973 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 02:36:20.410285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 7682
I20260620 02:36:20.428032 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 7019
I20260620 02:36:20.429250 7186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:20.528962 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.528416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.0.1:54882)
0620 02:36:20.528657 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:20.528766 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 02:36:20.530215 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:20.529814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42313 (local address 127.0.0.1:54896)
0620 02:36:20.529987 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:20.530076 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 02:36:20.547188 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 7019
I20260620 02:36:20.568778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 7118
I20260620 02:36:20.569923 7374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:20.694691 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 7118
2026-06-20T02:36:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12091 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:36:20 dist-test-slave-rpg6 krb5kdc[7955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:36:20 dist-test-slave-rpg6 krb5kdc[7955](info): set up 2 sockets
Jun 20 02:36:20 dist-test-slave-rpg6 krb5kdc[7955](info): commencing operation
krb5kdc: starting...
W20260620 02:36:21.828872 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:36:21 dist-test-slave-rpg6 krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:36:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 02:36:22.072983 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38467
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:22.433777 7968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:22.434165 7968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:22.434340 7968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:22.442201 7968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:22.442366 7968 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:36:22.442468 7968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:22.442579 7968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:22.442665 7968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:22.442777 7968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:36:22.453366 7968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38467
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:22.454802 7968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:22.456410 7968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:22.470008 7974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:22.470140 7973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:22.470155 7976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:22.470759 7968 server_base.cc:1034] running on GCE node
I20260620 02:36:22.471638 7968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:22.473263 7968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:22.474540 7968 hybrid_clock.cc:648] HybridClock initialized: now 1781922982474480 us; error 39 us; skew 500 ppm
I20260620 02:36:22.474916 7968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:22 dist-test-slave-rpg6 krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:22.487149 7968 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 02:36:22.489078 7968 webserver.cc:460] Webserver started at http://127.0.97.62:46563/ using document root <none> and password file <none>
I20260620 02:36:22.489907 7968 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:22.490005 7968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:22.490389 7968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:22.493635 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "73dee90f60c446cfab44b429aeac1f18"
format_stamp: "Formatted at 2026-06-20 02:36:22 on dist-test-slave-rpg6"
server_key: "373bb140446588fdca5574e8f95514ae"
server_key_iv: "fae421d994259286bfc7e05458715c3a"
server_key_version: "encryptionkey@0"
I20260620 02:36:22.494510 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "73dee90f60c446cfab44b429aeac1f18"
format_stamp: "Formatted at 2026-06-20 02:36:22 on dist-test-slave-rpg6"
server_key: "373bb140446588fdca5574e8f95514ae"
server_key_iv: "fae421d994259286bfc7e05458715c3a"
server_key_version: "encryptionkey@0"
I20260620 02:36:22.500415 7968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:36:22.504755 7983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:22.506494 7968 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:36:22.506718 7968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "73dee90f60c446cfab44b429aeac1f18"
format_stamp: "Formatted at 2026-06-20 02:36:22 on dist-test-slave-rpg6"
server_key: "373bb140446588fdca5574e8f95514ae"
server_key_iv: "fae421d994259286bfc7e05458715c3a"
server_key_version: "encryptionkey@0"
I20260620 02:36:22.507001 7968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:22.525638 7968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:22.530706 7968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:22.531060 7968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:22.555198 7968 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38467
I20260620 02:36:22.555219 8044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38467 every 8 connection(s)
I20260620 02:36:22.556740 7968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:22.557500 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 7968
I20260620 02:36:22.557746 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:36:22.558578 388 external_mini_cluster.cc:1428] Setting key 1d119b6a6e4fa2d7e07f5ec2d37f3e84
I20260620 02:36:22.561617 8045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:36:22 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:22.575642 8045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18: Bootstrap starting.
I20260620 02:36:22.580185 8045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:22.581460 8045 log.cc:824] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:22.584774 8045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18: No bootstrap required, opened a new log
I20260620 02:36:22.584982 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:22.560781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.0.97.62:38467)
0620 02:36:22.561684 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:22.561703 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:22.561758 (+ 55us) server_negotiation.cc:407] Connection header received
0620 02:36:22.564174 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:22.564204 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:22.564754 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:22.565290 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:22.566690 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:22.568918 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:22.570342 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:22.571420 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:22.571669 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:22.576201 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:22.576215 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:22.576238 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:22.576296 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:22.580386 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:22.581253 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:22.581263 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:22.581273 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:22.581499 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:22.581953 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:22.581961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:22.581966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:22.582639 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:22.582967 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:22.583431 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 02:36:22.583960 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":392,"threads_started":1}
I20260620 02:36:22.593227 8045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dee90f60c446cfab44b429aeac1f18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38467 } }
I20260620 02:36:22.593679 8045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:22.593791 8045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73dee90f60c446cfab44b429aeac1f18, State: Initialized, Role: FOLLOWER
I20260620 02:36:22.594378 8045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dee90f60c446cfab44b429aeac1f18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38467 } }
I20260620 02:36:22.594646 8045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:22.594794 8045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:22.594971 8045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:22.597561 8045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dee90f60c446cfab44b429aeac1f18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38467 } }
I20260620 02:36:22.598133 8045 leader_election.cc:302] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73dee90f60c446cfab44b429aeac1f18; no voters:
I20260620 02:36:22.598842 8045 leader_election.cc:288] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:36:22.599014 8050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:22.599916 8050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [term 1 LEADER]: Becoming Leader. State: Replica: 73dee90f60c446cfab44b429aeac1f18, State: Running, Role: LEADER
I20260620 02:36:22.600494 8050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dee90f60c446cfab44b429aeac1f18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38467 } }
I20260620 02:36:22.601003 8045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:22.603808 8052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73dee90f60c446cfab44b429aeac1f18. Latest consensus state: current_term: 1 leader_uuid: "73dee90f60c446cfab44b429aeac1f18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dee90f60c446cfab44b429aeac1f18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38467 } } }
I20260620 02:36:22.604449 8052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:22.604895 8051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73dee90f60c446cfab44b429aeac1f18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dee90f60c446cfab44b429aeac1f18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38467 } } }
I20260620 02:36:22.605198 8051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:22.607364 8059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:22.613072 8059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:22.623265 8059 catalog_manager.cc:1347] Generated new cluster ID: de283754d7d941be9b73be5057717d0e
I20260620 02:36:22.623412 8059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:22.633334 8059 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:36:22.635588 8059 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:22.645864 8059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18: Generated new TSK 0
I20260620 02:36:22.647128 8059 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 02:36:22.715978 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38467
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:23.076124 8071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:23.076444 8071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:23.076623 8071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:23.084460 8071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:23.084587 8071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:23.084753 8071 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:36:23.084843 8071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:36:23.095651 8071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:23.097177 8071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:23.098738 8071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:23.114763 8077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:23.114814 8076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:23.114816 8079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:23.116006 8071 server_base.cc:1034] running on GCE node
I20260620 02:36:23.116799 8071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:23.117898 8071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:23.119235 8071 hybrid_clock.cc:648] HybridClock initialized: now 1781922983119118 us; error 49 us; skew 500 ppm
I20260620 02:36:23.119625 8071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:23 dist-test-slave-rpg6 krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922983, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:23.123319 8071 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:36:23.125018 8071 webserver.cc:460] Webserver started at http://127.0.97.1:45239/ using document root <none> and password file <none>
I20260620 02:36:23.125823 8071 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:23.125970 8071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:23.126327 8071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:23.129711 8071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599"
format_stamp: "Formatted at 2026-06-20 02:36:23 on dist-test-slave-rpg6"
server_key: "b34af14f56be399385b542958712a85f"
server_key_iv: "72d83317a60f897ca9bd3926341731f5"
server_key_version: "encryptionkey@0"
I20260620 02:36:23.130638 8071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599"
format_stamp: "Formatted at 2026-06-20 02:36:23 on dist-test-slave-rpg6"
server_key: "b34af14f56be399385b542958712a85f"
server_key_iv: "72d83317a60f897ca9bd3926341731f5"
server_key_version: "encryptionkey@0"
I20260620 02:36:23.136754 8071 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 02:36:23.141254 8086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:23.142885 8071 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:36:23.143114 8071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599"
format_stamp: "Formatted at 2026-06-20 02:36:23 on dist-test-slave-rpg6"
server_key: "b34af14f56be399385b542958712a85f"
server_key_iv: "72d83317a60f897ca9bd3926341731f5"
server_key_version: "encryptionkey@0"
I20260620 02:36:23.143406 8071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:23.179139 8071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:23.184127 8071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:23.184470 8071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:23.187522 8071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:23.187697 8071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:23.187887 8071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:23.187999 8071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 02:36:23 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922983, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:23.215363 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:23.188770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45015 (local address 127.0.97.62:38467)
0620 02:36:23.189282 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:23.189287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:23.190575 (+ 1288us) server_negotiation.cc:407] Connection header received
0620 02:36:23.193190 (+ 2615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:23.193199 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:23.193320 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:23.193470 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:23.195213 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.196645 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.200971 (+ 4326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.201530 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.201693 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:23.208375 (+ 6682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:23.208403 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:23.208411 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:23.208476 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:23.211954 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:23.212889 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:23.212895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:23.212900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:23.213026 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:23.213500 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:23.213510 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:23.213512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:23.213791 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:23.213959 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:23.214750 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 02:36:23.214995 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":283,"threads_started":1}
I20260620 02:36:23.217161 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:23.189458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38467 (local address 127.0.97.1:45015)
0620 02:36:23.190378 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:23.190421 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:23.192720 (+ 2299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:23.193768 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:23.193777 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:23.194357 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:23.194987 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:23.195006 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.196942 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.196955 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:23.200666 (+ 3711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:23.200702 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.201894 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.201905 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:23.202402 (+ 497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:23.203571 (+ 1169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:23.203620 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:23.208008 (+ 4388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:23.212141 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:23.212153 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:23.212181 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:23.212733 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:23.213145 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:23.213150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:23.213154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:23.213372 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:23.213934 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:23.213942 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:23.214348 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 02:36:23.214635 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 02:36:23.215221 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":397,"threads_started":1}
I20260620 02:36:23.242125 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:23.225844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59381 (local address 127.0.97.62:38467)
0620 02:36:23.226040 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:23.226047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:23.226294 (+ 247us) server_negotiation.cc:407] Connection header received
0620 02:36:23.226588 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:23.226592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:23.226658 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:23.226781 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:23.227768 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.228785 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.230929 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.231560 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.231720 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:23.234951 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:23.234960 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:23.234967 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:23.235015 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:23.238266 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:23.239310 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:23.239318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:23.239322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:23.239490 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:23.240090 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:23.240095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:23.240098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:23.240518 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:23.240713 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:23.241491 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 02:36:23.241732 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:36:23.242229 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:23.225295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38467 (local address 127.0.97.1:59381)
0620 02:36:23.226161 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:23.226186 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:23.226423 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:23.226806 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:23.226811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:23.227317 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:23.227613 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:23.227623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.228939 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.228945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:23.230708 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:23.230731 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.232050 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.232061 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:23.232324 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:23.233256 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:23.233276 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:23.234615 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:23.238424 (+ 3809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:23.238429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:23.238437 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:23.239077 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:23.239633 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:23.239639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:23.239643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:23.239924 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:23.240880 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:23.240884 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:23.241165 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 02:36:23.241357 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 02:36:23.241758 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":738,"threads_started":1}
I20260620 02:36:23.247402 8071 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35629
I20260620 02:36:23.247464 8206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35629 every 8 connection(s)
I20260620 02:36:23.248785 8071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:36:23.251267 8207 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38467
I20260620 02:36:23.251381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 8071
I20260620 02:36:23.251578 8207 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:23.251641 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:36:23.252332 388 external_mini_cluster.cc:1428] Setting key 9960db657c9413b9af9f68bfad388275
I20260620 02:36:23.252462 8207 heartbeater.cc:511] Master 127.0.97.62:38467 requested a full tablet report, sending...
I20260620 02:36:23.254968 7999 ts_manager.cc:194] Registered new tserver with Master: 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629)
I20260620 02:36:23.257364 7999 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45015
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 02:36:23.309510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38467
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:23.664093 8211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:23.664407 8211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:23.664561 8211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:23.672777 8211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:23.672901 8211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:23.673040 8211 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:36:23.673141 8211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:36:23.683525 8211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:23.684947 8211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:23.686370 8211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:23.700945 8217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:23.701026 8216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:23.700945 8219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:23.701834 8211 server_base.cc:1034] running on GCE node
I20260620 02:36:23.702612 8211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:23.703647 8211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:23.704947 8211 hybrid_clock.cc:648] HybridClock initialized: now 1781922983704864 us; error 61 us; skew 500 ppm
I20260620 02:36:23.705430 8211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:23 dist-test-slave-rpg6 krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922983, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:23.709273 8211 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:36:23.711069 8211 webserver.cc:460] Webserver started at http://127.0.97.2:33943/ using document root <none> and password file <none>
I20260620 02:36:23.711802 8211 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:23.712019 8211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:23.712386 8211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:23.715555 8211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "8ce71c935e304e8c821000dc57f05a23"
format_stamp: "Formatted at 2026-06-20 02:36:23 on dist-test-slave-rpg6"
server_key: "18db21e0d9edbd8d66e5d8592410d941"
server_key_iv: "f93ce41419a55890d56052d67fbb140a"
server_key_version: "encryptionkey@0"
I20260620 02:36:23.716755 8211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8ce71c935e304e8c821000dc57f05a23"
format_stamp: "Formatted at 2026-06-20 02:36:23 on dist-test-slave-rpg6"
server_key: "18db21e0d9edbd8d66e5d8592410d941"
server_key_iv: "f93ce41419a55890d56052d67fbb140a"
server_key_version: "encryptionkey@0"
I20260620 02:36:23.723245 8211 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:36:23.727468 8226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:23.729250 8211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:36:23.729489 8211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "8ce71c935e304e8c821000dc57f05a23"
format_stamp: "Formatted at 2026-06-20 02:36:23 on dist-test-slave-rpg6"
server_key: "18db21e0d9edbd8d66e5d8592410d941"
server_key_iv: "f93ce41419a55890d56052d67fbb140a"
server_key_version: "encryptionkey@0"
I20260620 02:36:23.729772 8211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:23.746073 8211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:23.751492 8211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:23.751811 8211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:23.754835 8211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:23.754925 8211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:23.755121 8211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:23.755208 8211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:36:23 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922983, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:23.785652 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:23.755697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45347 (local address 127.0.97.62:38467)
0620 02:36:23.756219 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:23.756225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:23.757578 (+ 1353us) server_negotiation.cc:407] Connection header received
0620 02:36:23.760285 (+ 2707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:23.760291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:23.760396 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:23.760535 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:23.763267 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.764701 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.768537 (+ 3836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.769397 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.769604 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:23.776258 (+ 6654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:23.776268 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:23.776276 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:23.776335 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:23.780670 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:23.782353 (+ 1683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:23.782363 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:23.782369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:23.782587 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:23.783299 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:23.783304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:23.783309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:23.783821 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:23.784022 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:23.784970 (+ 948us) server_negotiation.cc:299] Negotiation successful
0620 02:36:23.785251 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":287,"threads_started":1}
I20260620 02:36:23.786931 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:23.756429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38467 (local address 127.0.97.2:45347)
0620 02:36:23.757345 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:23.757406 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:23.759897 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:23.760843 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:23.760852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:23.761826 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:23.762896 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:23.762936 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.764964 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.764975 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:23.768256 (+ 3281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:23.768289 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.769815 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.769829 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:23.770267 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:23.771375 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:23.771416 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:23.775859 (+ 4443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:23.780915 (+ 5056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:23.780934 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:23.780965 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:23.782130 (+ 1165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:23.782741 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:23.782752 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:23.782756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:23.783125 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:23.784019 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:23.784034 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:23.784556 (+ 522us) client_negotiation.cc:769] Sending connection context
0620 02:36:23.784856 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 02:36:23.785489 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":385,"threads_started":1}
I20260620 02:36:23.802011 8211 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34075
I20260620 02:36:23.802098 8346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34075 every 8 connection(s)
I20260620 02:36:23.803942 8211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:36:23.805013 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 8211
I20260620 02:36:23.805226 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:36:23.805775 388 external_mini_cluster.cc:1428] Setting key 32f10bcaf3c797a74ccff2730e3af36b
I20260620 02:36:23.806936 8347 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38467
I20260620 02:36:23.807307 8347 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:23.808598 8347 heartbeater.cc:511] Master 127.0.97.62:38467 requested a full tablet report, sending...
I20260620 02:36:23.810868 7999 ts_manager.cc:194] Registered new tserver with Master: 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2:34075)
I20260620 02:36:23.812706 7999 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45347
I20260620 02:36:23.813876 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:23.795196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48873 (local address 127.0.97.62:38467)
0620 02:36:23.795422 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:23.795432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:23.796206 (+ 774us) server_negotiation.cc:407] Connection header received
0620 02:36:23.796449 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:23.796455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:23.796549 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:23.796676 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:23.798597 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.799885 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.802654 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.803311 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.803477 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:23.806196 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:23.806204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:23.806208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:23.806254 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:23.810051 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:23.811113 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:23.811123 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:23.811142 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:23.811342 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:23.811865 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:23.811873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:23.811879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:23.812403 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:23.812534 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:23.813180 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 02:36:23.813460 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:36:23.815104 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:23.795160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38467 (local address 127.0.97.2:48873)
0620 02:36:23.796044 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:23.796079 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:23.796289 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:23.796913 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:23.796921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:23.797771 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:23.798337 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:23.798357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.800154 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.800160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:23.802460 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:23.802483 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:23.803573 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:23.803579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:23.803727 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:23.804535 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:23.804555 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:23.805905 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:23.810278 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:23.810287 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:23.810293 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:23.810960 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:23.811519 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:23.811526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:23.811533 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:23.811708 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:23.812638 (+ 930us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:23.812644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:23.812873 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 02:36:23.814073 (+ 1200us) client_negotiation.cc:240] Negotiation successful
0620 02:36:23.814490 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":535,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:36:23.863111 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38467
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:24.223093 8351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:24.223407 8351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:24.223562 8351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:24.231333 8351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:24.231472 8351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:24.231637 8351 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:36:24.231763 8351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:36:24.242405 8351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:24.243786 8351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:24.245395 8351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 02:36:24.261083 8207 heartbeater.cc:503] Master 127.0.97.62:38467 was elected leader, sending a full tablet report...
W20260620 02:36:24.263093 8357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:24.263152 8359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:24.263681 8356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:24.264214 8351 server_base.cc:1034] running on GCE node
I20260620 02:36:24.264952 8351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:24.266101 8351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:24.267379 8351 hybrid_clock.cc:648] HybridClock initialized: now 1781922984267300 us; error 57 us; skew 500 ppm
I20260620 02:36:24.267750 8351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:24 dist-test-slave-rpg6 krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922984, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:24.271520 8351 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:36:24.273283 8351 webserver.cc:460] Webserver started at http://127.0.97.3:44479/ using document root <none> and password file <none>
I20260620 02:36:24.274098 8351 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:24.274250 8351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:24.274608 8351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:24.278144 8351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368"
format_stamp: "Formatted at 2026-06-20 02:36:24 on dist-test-slave-rpg6"
server_key: "67dc5dbc55ae001f1a6c1e0b55c87fb1"
server_key_iv: "511e9b67362c1f6609df2f8727841b6b"
server_key_version: "encryptionkey@0"
I20260620 02:36:24.279074 8351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368"
format_stamp: "Formatted at 2026-06-20 02:36:24 on dist-test-slave-rpg6"
server_key: "67dc5dbc55ae001f1a6c1e0b55c87fb1"
server_key_iv: "511e9b67362c1f6609df2f8727841b6b"
server_key_version: "encryptionkey@0"
I20260620 02:36:24.285059 8351 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:36:24.290423 8366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:24.292914 8351 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 02:36:24.293170 8351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368"
format_stamp: "Formatted at 2026-06-20 02:36:24 on dist-test-slave-rpg6"
server_key: "67dc5dbc55ae001f1a6c1e0b55c87fb1"
server_key_iv: "511e9b67362c1f6609df2f8727841b6b"
server_key_version: "encryptionkey@0"
I20260620 02:36:24.293530 8351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:24.346489 8351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:24.351392 8351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:24.351709 8351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:24.354645 8351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:24.354807 8351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:24.354957 8351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:24.355028 8351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:36:24 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922984, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:24.384790 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.355648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58761 (local address 127.0.97.62:38467)
0620 02:36:24.356162 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:24.356167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:24.357633 (+ 1466us) server_negotiation.cc:407] Connection header received
0620 02:36:24.360113 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:24.360123 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:24.360251 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:24.360429 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:24.363005 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.364601 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.368479 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.369019 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.369228 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.375395 (+ 6167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:24.375401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:24.375405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:24.375444 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:24.380012 (+ 4568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:24.381246 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:24.381255 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:24.381261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:24.381460 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:24.382038 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:24.382043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:24.382045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:24.382440 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:24.382602 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:24.383815 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0620 02:36:24.384168 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":279,"threads_started":1}
I20260620 02:36:24.385766 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.356438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38467 (local address 127.0.97.3:58761)
0620 02:36:24.357411 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:24.357478 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:24.359623 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:24.360768 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:24.360778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:24.361656 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:24.362750 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.362779 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.364870 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.364883 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.368203 (+ 3320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.368242 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.369420 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.369432 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.369887 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.370963 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:24.371012 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:24.375065 (+ 4053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:24.380251 (+ 5186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:24.380264 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:24.380291 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:24.381068 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:24.381622 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:24.381628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:24.381631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:24.381889 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:24.382666 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:24.382673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:24.383289 (+ 616us) client_negotiation.cc:769] Sending connection context
0620 02:36:24.383653 (+ 364us) client_negotiation.cc:240] Negotiation successful
0620 02:36:24.384377 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":389,"threads_started":1}
I20260620 02:36:24.413093 8351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45141
I20260620 02:36:24.413105 8486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45141 every 8 connection(s)
I20260620 02:36:24.414476 8351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:36:24.416762 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.400010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38467 (local address 127.0.97.3:60597)
0620 02:36:24.402437 (+ 2427us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:24.402463 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:24.402687 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:24.403156 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:24.403160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:24.404029 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:24.404632 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.404646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.406100 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.406106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.407548 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.407564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.408563 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.408569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.408747 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.409443 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:24.409462 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:24.410263 (+ 801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:24.413923 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:24.413929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:24.413933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:24.414382 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:24.414813 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:24.414818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:24.414822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:24.414955 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:24.415661 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:24.415666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:24.415930 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 02:36:24.416070 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 02:36:24.416405 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2183,"thread_start_us":1956,"threads_started":1}
I20260620 02:36:24.417527 8487 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38467
I20260620 02:36:24.417797 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.401273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60597 (local address 127.0.97.62:38467)
0620 02:36:24.401689 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:24.401698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:24.402596 (+ 898us) server_negotiation.cc:407] Connection header received
0620 02:36:24.402878 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:24.402884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:24.402996 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:24.403128 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:24.404846 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.405860 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.407720 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.408351 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.408501 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.410505 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:24.410511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:24.410514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:24.410549 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:24.413751 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:24.414530 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:24.414537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:24.414540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:24.414692 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:24.415086 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:24.415092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:24.415095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:24.415494 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:24.417090 (+ 1596us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:24.417220 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 02:36:24.417502 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:36:24.418371 8487 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:24.418668 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 8351
I20260620 02:36:24.418861 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:36:24.419394 8487 heartbeater.cc:511] Master 127.0.97.62:38467 requested a full tablet report, sending...
I20260620 02:36:24.419536 388 external_mini_cluster.cc:1428] Setting key 4df677967f842a35304634217fe2559b
I20260620 02:36:24.421926 7999 ts_manager.cc:194] Registered new tserver with Master: 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141)
I20260620 02:36:24.423005 7999 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58761
I20260620 02:36:24.423952 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:36:24 dist-test-slave-rpg6 krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:36:24 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:24 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:36:24.476570 8375 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:24.465318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.0.97.62:38467)
0620 02:36:24.465587 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:24.465596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:24.465877 (+ 281us) server_negotiation.cc:407] Connection header received
0620 02:36:24.466097 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:24.466103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:24.466216 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:24.466335 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:24.467685 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.468495 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.469805 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.470206 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.470346 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.476211 (+ 5865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43386: BlockingRecv error: recv got EOF from 127.0.0.1:43386 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 20 02:36:24 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:24.495103 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.479729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.0.97.62:38467)
0620 02:36:24.480031 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:24.480037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:24.480213 (+ 176us) server_negotiation.cc:407] Connection header received
0620 02:36:24.480356 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:24.480360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:24.480424 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:24.480514 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:24.481738 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.482453 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.484293 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.484739 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.484850 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.489086 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:24.489096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:24.489105 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:24.489148 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:24.492370 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:24.493169 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:24.493174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:24.493176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:24.493315 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:24.493673 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:24.493676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:24.493679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:24.493959 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:24.494177 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:24.494534 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 02:36:24.494715 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260620 02:36:24.500027 7999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:36:24.507870 7999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:36:24.532491 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.522057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.0.97.3:45141)
0620 02:36:24.523714 (+ 1657us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:24.523723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:24.523759 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:36:24.523907 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:24.523920 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:24.524232 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:24.524458 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:24.525183 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.526950 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.529630 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.531214 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.531362 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.531623 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:24.531933 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 02:36:24.532121 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1443,"thread_start_us":1244,"threads_started":1}
I20260620 02:36:24.532554 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.521767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45141 (local address 127.0.0.1:39028)
0620 02:36:24.522706 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:24.522759 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:24.522944 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:24.524458 (+ 1514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:24.524466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:24.524497 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:24.524942 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.524958 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.527119 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.527124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.529430 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.529444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.531419 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.531425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.531626 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.531692 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 02:36:24.531870 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 02:36:24.532085 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":1,"thread_start_us":492,"threads_started":1}
I20260620 02:36:24.533192 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.522317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34075 (local address 127.0.0.1:48976)
0620 02:36:24.524307 (+ 1990us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:24.524331 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:24.524473 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:24.525595 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:24.525599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:24.525626 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:24.526145 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.526162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.527950 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.527956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.530148 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.530163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.532270 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.532278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.532458 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.532504 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 02:36:24.532606 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 02:36:24.532767 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1837,"mutex_wait_us":39,"thread_start_us":1669,"threads_started":1}
I20260620 02:36:24.533826 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.523319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35629 (local address 127.0.0.1:46192)
0620 02:36:24.525596 (+ 2277us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:24.525614 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:24.525760 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:24.526622 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:24.526626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:24.526649 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:24.527038 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.527053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.528918 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.528926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.530883 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.530895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.532981 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.532988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.533177 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.533217 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:36:24.533356 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 02:36:24.533498 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2149,"mutex_wait_us":1,"thread_start_us":2008,"threads_started":1}
I20260620 02:36:24.534073 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.522902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.0.97.2:34075)
0620 02:36:24.524742 (+ 1840us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:24.524753 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:24.524786 (+ 33us) server_negotiation.cc:407] Connection header received
0620 02:36:24.524894 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:24.524901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:24.525140 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:24.525381 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:24.526345 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.527805 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.530358 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.531752 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.531923 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.532105 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:24.533649 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0620 02:36:24.533788 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":720,"threads_started":1}
I20260620 02:36:24.534600 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.523766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.0.97.1:35629)
0620 02:36:24.525158 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:24.525165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:24.525891 (+ 726us) server_negotiation.cc:407] Connection header received
0620 02:36:24.526000 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:24.526007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:24.526212 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:24.526432 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:24.527231 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.528477 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.531059 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.532497 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.532634 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.532838 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:24.534036 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0620 02:36:24.534230 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1234,"thread_start_us":1095,"threads_started":1}
I20260620 02:36:24.537293 8417 tablet_service.cc:1467] Processing CreateTablet for tablet 3d114c5566834d80a571ec0db7af5691 (DEFAULT_TABLE table=test-table [id=31d046ae7a9e485994efde8c605fe125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:24.537530 8277 tablet_service.cc:1467] Processing CreateTablet for tablet 3d114c5566834d80a571ec0db7af5691 (DEFAULT_TABLE table=test-table [id=31d046ae7a9e485994efde8c605fe125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:24.538987 8277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d114c5566834d80a571ec0db7af5691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:24.538991 8417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d114c5566834d80a571ec0db7af5691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:24.538790 8137 tablet_service.cc:1467] Processing CreateTablet for tablet 3d114c5566834d80a571ec0db7af5691 (DEFAULT_TABLE table=test-table [id=31d046ae7a9e485994efde8c605fe125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:24.540254 8137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d114c5566834d80a571ec0db7af5691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:24.553429 8511 tablet_bootstrap.cc:492] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Bootstrap starting.
I20260620 02:36:24.558320 8513 tablet_bootstrap.cc:492] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Bootstrap starting.
I20260620 02:36:24.558808 8511 tablet_bootstrap.cc:654] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:24.560345 8511 log.cc:824] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:24.563396 8513 tablet_bootstrap.cc:654] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:24.566617 8512 tablet_bootstrap.cc:492] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23: Bootstrap starting.
I20260620 02:36:24.569897 8511 tablet_bootstrap.cc:492] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: No bootstrap required, opened a new log
I20260620 02:36:24.570194 8511 ts_tablet_manager.cc:1397] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260620 02:36:24.570644 8512 tablet_bootstrap.cc:654] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:24.571707 8512 log.cc:824] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:24.572688 8513 log.cc:824] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:24.581455 8511 raft_consensus.cc:348] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } }
I20260620 02:36:24.581997 8511 raft_consensus.cc:374] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:24.582088 8511 raft_consensus.cc:729] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fb1fb2a00415d8602e7b8ed8f8368, State: Initialized, Role: FOLLOWER
I20260620 02:36:24.583355 8511 consensus_queue.cc:260] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } }
I20260620 02:36:24.584614 8487 heartbeater.cc:503] Master 127.0.97.62:38467 was elected leader, sending a full tablet report...
I20260620 02:36:24.585402 8511 ts_tablet_manager.cc:1428] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260620 02:36:24.585690 8513 tablet_bootstrap.cc:492] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: No bootstrap required, opened a new log
I20260620 02:36:24.586010 8513 ts_tablet_manager.cc:1397] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.009s
I20260620 02:36:24.587069 8512 tablet_bootstrap.cc:492] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23: No bootstrap required, opened a new log
I20260620 02:36:24.587383 8512 ts_tablet_manager.cc:1397] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.007s
I20260620 02:36:24.595815 8512 raft_consensus.cc:348] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } }
I20260620 02:36:24.595784 8513 raft_consensus.cc:348] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } }
I20260620 02:36:24.596303 8512 raft_consensus.cc:374] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:24.596304 8513 raft_consensus.cc:374] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:24.596423 8512 raft_consensus.cc:729] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce71c935e304e8c821000dc57f05a23, State: Initialized, Role: FOLLOWER
I20260620 02:36:24.596451 8513 raft_consensus.cc:729] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8e0d990ec84fb7be8fc8e9b78c0599, State: Initialized, Role: FOLLOWER
I20260620 02:36:24.597056 8512 consensus_queue.cc:260] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } }
I20260620 02:36:24.597060 8513 consensus_queue.cc:260] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } }
I20260620 02:36:24.598964 8513 ts_tablet_manager.cc:1428] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 02:36:24.599231 8512 ts_tablet_manager.cc:1428] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260620 02:36:24.599499 8347 heartbeater.cc:503] Master 127.0.97.62:38467 was elected leader, sending a full tablet report...
I20260620 02:36:24.658736 8518 raft_consensus.cc:482] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:36:24.659102 8518 raft_consensus.cc:504] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } }
I20260620 02:36:24.661046 8518 leader_election.cc:288] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141), 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629)
W20260620 02:36:24.666438 8488 tablet.cc:2369] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:36:24.669155 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.661491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45141 (local address 127.0.97.2:53397)
0620 02:36:24.662315 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:24.662357 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:24.662542 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:24.663108 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:24.663112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:24.663134 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:24.663559 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.663571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.664801 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.664807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.666469 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.666482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.668292 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.668299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.668478 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.668516 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:36:24.668626 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 02:36:24.668803 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":464,"threads_started":1}
I20260620 02:36:24.669229 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.661830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53397 (local address 127.0.97.3:45141)
0620 02:36:24.662130 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:24.662137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:24.662488 (+ 351us) server_negotiation.cc:407] Connection header received
0620 02:36:24.662738 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:24.662745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:24.662829 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:24.662898 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:24.663716 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.664665 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.666738 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.667883 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.668006 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.668124 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:24.668764 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 02:36:24.668957 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260620 02:36:24.669788 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.662717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41365 (local address 127.0.97.1:35629)
0620 02:36:24.662909 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:24.662916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:24.663223 (+ 307us) server_negotiation.cc:407] Connection header received
0620 02:36:24.663364 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:24.663369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:24.663458 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:24.663571 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:24.664273 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.665197 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.667384 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.668640 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.668804 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.668899 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:24.669377 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 02:36:24.669532 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:36:24.670552 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.662407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35629 (local address 127.0.97.2:41365)
0620 02:36:24.663106 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:24.663122 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:24.663236 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:24.663592 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:24.663596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:24.663614 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:24.664084 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.664095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.665351 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.665355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.667199 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:24.667213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.668924 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.668930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:24.669113 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.669180 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 02:36:24.669980 (+ 800us) client_negotiation.cc:240] Negotiation successful
0620 02:36:24.670149 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":1,"thread_start_us":372,"threads_started":1}
I20260620 02:36:24.670603 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d114c5566834d80a571ec0db7af5691" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" is_pre_election: true
I20260620 02:36:24.671262 8437 raft_consensus.cc:2454] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 0.
I20260620 02:36:24.671710 8162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d114c5566834d80a571ec0db7af5691" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" is_pre_election: true
I20260620 02:36:24.672071 8229 leader_election.cc:302] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b0fb1fb2a00415d8602e7b8ed8f8368, 8ce71c935e304e8c821000dc57f05a23; no voters:
I20260620 02:36:24.672384 8162 raft_consensus.cc:2454] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 0.
I20260620 02:36:24.672559 8518 raft_consensus.cc:2781] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:36:24.672729 8518 raft_consensus.cc:482] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:36:24.672861 8518 raft_consensus.cc:3037] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:24.675285 8518 raft_consensus.cc:504] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } }
I20260620 02:36:24.675966 8518 leader_election.cc:288] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 election: Requested vote from peers 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141), 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629)
I20260620 02:36:24.676594 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d114c5566834d80a571ec0db7af5691" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368"
I20260620 02:36:24.676644 8162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d114c5566834d80a571ec0db7af5691" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599"
I20260620 02:36:24.676887 8437 raft_consensus.cc:3037] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:24.676947 8162 raft_consensus.cc:3037] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:24.679286 8437 raft_consensus.cc:2454] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 1.
I20260620 02:36:24.679301 8162 raft_consensus.cc:2454] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 1.
I20260620 02:36:24.679792 8229 leader_election.cc:302] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b8e0d990ec84fb7be8fc8e9b78c0599, 8ce71c935e304e8c821000dc57f05a23; no voters:
I20260620 02:36:24.680202 8518 raft_consensus.cc:2781] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:24.680704 8518 raft_consensus.cc:686] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 1 LEADER]: Becoming Leader. State: Replica: 8ce71c935e304e8c821000dc57f05a23, State: Running, Role: LEADER
I20260620 02:36:24.681280 8518 consensus_queue.cc:237] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } }
I20260620 02:36:24.686452 8000 catalog_manager.cc:5495] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8ce71c935e304e8c821000dc57f05a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } health_report { overall_health: HEALTHY } } }
I20260620 02:36:24.737318 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:24.731616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.0.97.2:34075)
0620 02:36:24.731874 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:24.731880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:24.731904 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:36:24.731996 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:24.732000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:24.732088 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:24.732268 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:36:24.732621 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.733397 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.734956 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:24.735406 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:24.735552 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:24.736099 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:36:24.736679 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:36:24.736847 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:24.736931 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 02:36:24.737015 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:36:24.746057 8162 raft_consensus.cc:1261] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Refusing update from remote peer 8ce71c935e304e8c821000dc57f05a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:24.746057 8437 raft_consensus.cc:1261] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Refusing update from remote peer 8ce71c935e304e8c821000dc57f05a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:24.747032 8518 consensus_queue.cc:1035] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:24.747373 8522 consensus_queue.cc:1035] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:36:24.751788 8208 tablet.cc:2369] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:36:24.761938 8526 mvcc.cc:205] Tried to move back new op lower bound from 7298756545509171200 to 7298756545259040768. Current Snapshot: MvccSnapshot[applied={T|T < 7298756545509171200}]
I20260620 02:36:24.777220 7999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:24.777552 7999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:24.781546 7999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18: Sending DeleteTablet for 3 replicas of tablet 3d114c5566834d80a571ec0db7af5691
I20260620 02:36:24.782968 8277 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d114c5566834d80a571ec0db7af5691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48976
I20260620 02:36:24.783159 8137 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d114c5566834d80a571ec0db7af5691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46192
I20260620 02:36:24.783790 8417 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d114c5566834d80a571ec0db7af5691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39028
I20260620 02:36:24.784770 7999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:36:24.785954 7999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:36:24.790894 8538 tablet_replica.cc:330] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: stopping tablet replica
I20260620 02:36:24.791500 8538 raft_consensus.cc:2229] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:24.790391 8537 tablet_replica.cc:330] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: stopping tablet replica
I20260620 02:36:24.792204 8537 raft_consensus.cc:2229] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:24.792810 8537 raft_consensus.cc:2258] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:24.793272 8538 raft_consensus.cc:2258] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:24.795773 8136 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5a0462a0624ef9bd702a1ec3f3eb05 (DEFAULT_TABLE table=test-table [id=ebbdaea9736240e688090570ef44055c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:24.796258 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5a0462a0624ef9bd702a1ec3f3eb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:24.796236 8276 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5a0462a0624ef9bd702a1ec3f3eb05 (DEFAULT_TABLE table=test-table [id=ebbdaea9736240e688090570ef44055c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:24.796646 8536 tablet_replica.cc:330] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23: stopping tablet replica
I20260620 02:36:24.796816 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5a0462a0624ef9bd702a1ec3f3eb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:24.797103 8536 raft_consensus.cc:2229] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:36:24.797919 8536 raft_consensus.cc:2258] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:24.799526 8417 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5a0462a0624ef9bd702a1ec3f3eb05 (DEFAULT_TABLE table=test-table [id=ebbdaea9736240e688090570ef44055c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:24.800092 8417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5a0462a0624ef9bd702a1ec3f3eb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:24.800446 8537 ts_tablet_manager.cc:1905] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:24.803021 8538 ts_tablet_manager.cc:1905] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:24.803390 8513 tablet_bootstrap.cc:492] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Bootstrap starting.
I20260620 02:36:24.805226 8513 tablet_bootstrap.cc:654] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:24.808085 8511 tablet_bootstrap.cc:492] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Bootstrap starting.
I20260620 02:36:24.808499 8537 ts_tablet_manager.cc:1918] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:24.808678 8537 log.cc:1196] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/3d114c5566834d80a571ec0db7af5691
I20260620 02:36:24.809865 8511 tablet_bootstrap.cc:654] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:24.810299 8537 ts_tablet_manager.cc:1939] T 3d114c5566834d80a571ec0db7af5691 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting consensus metadata
I20260620 02:36:24.811498 8536 ts_tablet_manager.cc:1905] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:24.812444 7986 catalog_manager.cc:4841] TS 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629): tablet 3d114c5566834d80a571ec0db7af5691 (table test-table [id=31d046ae7a9e485994efde8c605fe125]) successfully deleted
I20260620 02:36:24.814807 8538 ts_tablet_manager.cc:1918] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:24.814991 8538 log.cc:1196] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/3d114c5566834d80a571ec0db7af5691
I20260620 02:36:24.815346 8511 tablet_bootstrap.cc:492] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368: No bootstrap required, opened a new log
I20260620 02:36:24.815522 8511 ts_tablet_manager.cc:1397] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260620 02:36:24.815721 8538 ts_tablet_manager.cc:1939] T 3d114c5566834d80a571ec0db7af5691 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting consensus metadata
I20260620 02:36:24.816460 8511 raft_consensus.cc:348] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:24.816695 8511 raft_consensus.cc:374] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:24.816792 8511 raft_consensus.cc:729] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fb1fb2a00415d8602e7b8ed8f8368, State: Initialized, Role: FOLLOWER
I20260620 02:36:24.817082 8511 consensus_queue.cc:260] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:24.819094 7986 catalog_manager.cc:4841] TS 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141): tablet 3d114c5566834d80a571ec0db7af5691 (table test-table [id=31d046ae7a9e485994efde8c605fe125]) successfully deleted
I20260620 02:36:24.819679 8511 ts_tablet_manager.cc:1428] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 02:36:24.821075 8513 tablet_bootstrap.cc:492] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599: No bootstrap required, opened a new log
I20260620 02:36:24.821249 8513 ts_tablet_manager.cc:1397] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20260620 02:36:24.822050 8513 raft_consensus.cc:348] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:24.822288 8513 raft_consensus.cc:374] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:24.822381 8513 raft_consensus.cc:729] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8e0d990ec84fb7be8fc8e9b78c0599, State: Initialized, Role: FOLLOWER
I20260620 02:36:24.822742 8512 tablet_bootstrap.cc:492] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23: Bootstrap starting.
I20260620 02:36:24.822702 8513 consensus_queue.cc:260] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:24.823851 8513 ts_tablet_manager.cc:1428] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 02:36:24.824657 8512 tablet_bootstrap.cc:654] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:24.827178 8536 ts_tablet_manager.cc:1918] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:24.827354 8536 log.cc:1196] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/3d114c5566834d80a571ec0db7af5691
I20260620 02:36:24.827919 8536 ts_tablet_manager.cc:1939] T 3d114c5566834d80a571ec0db7af5691 P 8ce71c935e304e8c821000dc57f05a23: Deleting consensus metadata
I20260620 02:36:24.829782 7987 catalog_manager.cc:4841] TS 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2:34075): tablet 3d114c5566834d80a571ec0db7af5691 (table test-table [id=31d046ae7a9e485994efde8c605fe125]) successfully deleted
I20260620 02:36:24.839239 8512 tablet_bootstrap.cc:492] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23: No bootstrap required, opened a new log
I20260620 02:36:24.839459 8512 ts_tablet_manager.cc:1397] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260620 02:36:24.840370 8512 raft_consensus.cc:348] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:24.840600 8512 raft_consensus.cc:374] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:24.840745 8512 raft_consensus.cc:729] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce71c935e304e8c821000dc57f05a23, State: Initialized, Role: FOLLOWER
I20260620 02:36:24.841027 8512 consensus_queue.cc:260] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:24.841660 8512 ts_tablet_manager.cc:1428] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:36:25.044661 8519 raft_consensus.cc:482] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:36:25.044952 8519 raft_consensus.cc:504] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.047008 8519 leader_election.cc:288] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2:34075), 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141)
I20260620 02:36:25.055557 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.047997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59827 (local address 127.0.97.3:45141)
0620 02:36:25.048259 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.048265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.049015 (+ 750us) server_negotiation.cc:407] Connection header received
0620 02:36:25.049177 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.049184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.049262 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.049370 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:25.050140 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.051173 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.053319 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.054438 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.054589 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.054716 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:25.055091 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 02:36:25.055252 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 02:36:25.055935 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.047390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45141 (local address 127.0.97.1:59827)
0620 02:36:25.048886 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:25.048905 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:25.049048 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:25.049421 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:25.049429 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:25.049460 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:25.049877 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.049889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.051309 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.051314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.053122 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.053137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.054700 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.054708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.054883 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.054922 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:36:25.055035 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 02:36:25.055199 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"mutex_wait_us":129,"thread_start_us":1127,"threads_started":1}
I20260620 02:36:25.056762 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5a0462a0624ef9bd702a1ec3f3eb05" candidate_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" is_pre_election: true
I20260620 02:36:25.056995 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.047631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51235 (local address 127.0.97.2:34075)
0620 02:36:25.047846 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.047853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.048310 (+ 457us) server_negotiation.cc:407] Connection header received
0620 02:36:25.048662 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.048668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.048747 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.048822 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:25.049699 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.051035 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.053761 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.055070 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.055228 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.055350 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:25.056493 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0620 02:36:25.056672 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:36:25.057020 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.047377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34075 (local address 127.0.97.1:51235)
0620 02:36:25.048178 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:25.048205 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:25.048467 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:25.049036 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:25.049044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:25.049075 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:25.049482 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.049497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.051208 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.051214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.053560 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.053577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.055941 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.055946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.056205 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.056240 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:36:25.056423 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 02:36:25.056635 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":101,"thread_start_us":436,"threads_started":1}
I20260620 02:36:25.057060 8437 raft_consensus.cc:2454] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b8e0d990ec84fb7be8fc8e9b78c0599 in term 0.
I20260620 02:36:25.057866 8089 leader_election.cc:302] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b8e0d990ec84fb7be8fc8e9b78c0599, 8b0fb1fb2a00415d8602e7b8ed8f8368; no voters:
I20260620 02:36:25.057953 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5a0462a0624ef9bd702a1ec3f3eb05" candidate_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce71c935e304e8c821000dc57f05a23" is_pre_election: true
I20260620 02:36:25.058382 8519 raft_consensus.cc:2781] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:36:25.058527 8519 raft_consensus.cc:482] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:36:25.058633 8519 raft_consensus.cc:3037] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:25.058552 8297 raft_consensus.cc:2454] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b8e0d990ec84fb7be8fc8e9b78c0599 in term 0.
I20260620 02:36:25.062002 8519 raft_consensus.cc:504] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.062906 8519 leader_election.cc:288] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [CANDIDATE]: Term 1 election: Requested vote from peers 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2:34075), 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141)
I20260620 02:36:25.063534 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5a0462a0624ef9bd702a1ec3f3eb05" candidate_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368"
I20260620 02:36:25.063539 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5a0462a0624ef9bd702a1ec3f3eb05" candidate_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce71c935e304e8c821000dc57f05a23"
I20260620 02:36:25.063771 8297 raft_consensus.cc:3037] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:25.063764 8437 raft_consensus.cc:3037] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:25.065949 8437 raft_consensus.cc:2454] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b8e0d990ec84fb7be8fc8e9b78c0599 in term 1.
I20260620 02:36:25.066637 8089 leader_election.cc:302] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b8e0d990ec84fb7be8fc8e9b78c0599, 8b0fb1fb2a00415d8602e7b8ed8f8368; no voters:
I20260620 02:36:25.067054 8519 raft_consensus.cc:2781] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:25.067642 8297 raft_consensus.cc:2454] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b8e0d990ec84fb7be8fc8e9b78c0599 in term 1.
I20260620 02:36:25.067869 8519 raft_consensus.cc:686] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 LEADER]: Becoming Leader. State: Replica: 4b8e0d990ec84fb7be8fc8e9b78c0599, State: Running, Role: LEADER
I20260620 02:36:25.068594 8519 consensus_queue.cc:237] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
W20260620 02:36:25.069399 8348 tablet.cc:2369] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:36:25.073073 7999 catalog_manager.cc:5495] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } health_report { overall_health: UNKNOWN } } }
Jun 20 02:36:25 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922982, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:25.148262 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.130403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38467 (local address 127.0.0.1:43392)
0620 02:36:25.131164 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:25.131188 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:25.131356 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:25.131815 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:25.131819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:25.132546 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:25.132764 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.132774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.134417 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.134422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.135663 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.135679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.136746 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.136751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.136900 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.137743 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:25.137762 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:25.140555 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:25.145120 (+ 4565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:25.145138 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:25.145146 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:25.145632 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:25.146141 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:25.146145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:25.146148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:25.146371 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:25.147135 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:25.147147 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:25.147346 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 02:36:25.147511 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 02:36:25.147884 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":399,"threads_started":1}
I20260620 02:36:25.148530 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.130497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.0.97.62:38467)
0620 02:36:25.131546 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.131551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.131568 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:36:25.131612 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.131615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.131670 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.131792 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:25.133019 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.134244 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.135889 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.136528 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.136679 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.140835 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:25.140843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:25.140849 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:25.140893 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:25.144886 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:25.145810 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:25.145816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:25.145820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:25.145985 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:25.146516 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:25.146522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:25.146526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:25.146930 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:25.147162 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:25.147657 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 02:36:25.148031 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":908,"thread_start_us":674,"threads_started":1}
I20260620 02:36:25.154073 7999 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43392:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:36:25.155287 7999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:36:25.163544 8136 tablet_service.cc:1467] Processing CreateTablet for tablet db2e82804bb54a0fb8214814af88cdf9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3e657c606874a6b937db507e188b1cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:36:25.163602 8276 tablet_service.cc:1467] Processing CreateTablet for tablet db2e82804bb54a0fb8214814af88cdf9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3e657c606874a6b937db507e188b1cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:36:25.163717 8417 tablet_service.cc:1467] Processing CreateTablet for tablet db2e82804bb54a0fb8214814af88cdf9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3e657c606874a6b937db507e188b1cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:36:25.164194 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2e82804bb54a0fb8214814af88cdf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:25.164211 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2e82804bb54a0fb8214814af88cdf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:25.164227 8417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2e82804bb54a0fb8214814af88cdf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:25.172638 8512 tablet_bootstrap.cc:492] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: Bootstrap starting.
I20260620 02:36:25.172724 8513 tablet_bootstrap.cc:492] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Bootstrap starting.
I20260620 02:36:25.174610 8512 tablet_bootstrap.cc:654] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:25.174626 8513 tablet_bootstrap.cc:654] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:25.175812 8511 tablet_bootstrap.cc:492] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Bootstrap starting.
I20260620 02:36:25.178349 8511 tablet_bootstrap.cc:654] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:25.183629 8511 tablet_bootstrap.cc:492] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368: No bootstrap required, opened a new log
I20260620 02:36:25.185483 8511 ts_tablet_manager.cc:1397] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 02:36:25.186404 8511 raft_consensus.cc:348] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.186673 8513 tablet_bootstrap.cc:492] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599: No bootstrap required, opened a new log
I20260620 02:36:25.186678 8511 raft_consensus.cc:374] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:25.186785 8511 raft_consensus.cc:729] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fb1fb2a00415d8602e7b8ed8f8368, State: Initialized, Role: FOLLOWER
I20260620 02:36:25.186892 8513 ts_tablet_manager.cc:1397] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260620 02:36:25.187119 8511 consensus_queue.cc:260] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.187590 8517 tablet_replica.cc:439] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } }
I20260620 02:36:25.187664 8513 raft_consensus.cc:348] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.187925 8517 tablet_replica.cc:442] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:36:25.187920 8513 raft_consensus.cc:374] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:25.187995 8513 raft_consensus.cc:729] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8e0d990ec84fb7be8fc8e9b78c0599, State: Initialized, Role: FOLLOWER
I20260620 02:36:25.188235 8513 consensus_queue.cc:260] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.188784 8513 ts_tablet_manager.cc:1428] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:36:25.188877 8519 tablet_replica.cc:439] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } }
I20260620 02:36:25.189174 8519 tablet_replica.cc:442] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:36:25.192476 8511 ts_tablet_manager.cc:1428] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 02:36:25.195672 8512 tablet_bootstrap.cc:492] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: No bootstrap required, opened a new log
I20260620 02:36:25.195922 8512 ts_tablet_manager.cc:1397] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.007s
I20260620 02:36:25.196827 8512 raft_consensus.cc:348] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.197116 8512 raft_consensus.cc:374] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:25.197216 8512 raft_consensus.cc:729] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce71c935e304e8c821000dc57f05a23, State: Initialized, Role: FOLLOWER
I20260620 02:36:25.197489 8512 consensus_queue.cc:260] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.198339 8512 ts_tablet_manager.cc:1428] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:36:25.198077 8522 tablet_replica.cc:439] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } }
I20260620 02:36:25.198529 8522 tablet_replica.cc:442] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:36:25.403491 8522 raft_consensus.cc:482] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:36:25.403728 8522 raft_consensus.cc:504] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.404526 8522 leader_election.cc:288] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629), 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141)
I20260620 02:36:25.405296 8162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2e82804bb54a0fb8214814af88cdf9" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" is_pre_election: true
I20260620 02:36:25.405325 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2e82804bb54a0fb8214814af88cdf9" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" is_pre_election: true
I20260620 02:36:25.405578 8162 raft_consensus.cc:2454] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 0.
I20260620 02:36:25.405694 8437 raft_consensus.cc:2454] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 0.
I20260620 02:36:25.406114 8229 leader_election.cc:302] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b8e0d990ec84fb7be8fc8e9b78c0599, 8ce71c935e304e8c821000dc57f05a23; no voters:
I20260620 02:36:25.406400 8522 raft_consensus.cc:2781] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:36:25.406514 8522 raft_consensus.cc:482] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:36:25.406625 8522 raft_consensus.cc:3037] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:25.409005 8522 raft_consensus.cc:504] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.409603 8522 leader_election.cc:288] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 election: Requested vote from peers 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629), 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141)
I20260620 02:36:25.410354 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2e82804bb54a0fb8214814af88cdf9" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368"
I20260620 02:36:25.410601 8437 raft_consensus.cc:3037] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:25.412986 8437 raft_consensus.cc:2454] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 1.
I20260620 02:36:25.413516 8229 leader_election.cc:302] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b0fb1fb2a00415d8602e7b8ed8f8368, 8ce71c935e304e8c821000dc57f05a23; no voters:
I20260620 02:36:25.413841 8522 raft_consensus.cc:2781] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:25.414254 8522 raft_consensus.cc:686] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [term 1 LEADER]: Becoming Leader. State: Replica: 8ce71c935e304e8c821000dc57f05a23, State: Running, Role: LEADER
I20260620 02:36:25.414531 8522 consensus_queue.cc:237] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.416126 8162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2e82804bb54a0fb8214814af88cdf9" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599"
I20260620 02:36:25.416083 8556 tablet_replica.cc:439] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: TxnStatusTablet state changed. Reason: New leader 8ce71c935e304e8c821000dc57f05a23. Latest consensus state: current_term: 1 leader_uuid: "8ce71c935e304e8c821000dc57f05a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } }
I20260620 02:36:25.416360 8162 raft_consensus.cc:3037] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:25.416445 8556 tablet_replica.cc:442] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:36:25.417054 8558 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:36:25.417289 8558 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:36:25.420284 7999 catalog_manager.cc:5495] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8ce71c935e304e8c821000dc57f05a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20260620 02:36:25.421026 8162 raft_consensus.cc:2454] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 1.
I20260620 02:36:25.491043 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.484783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.0.97.2:34075)
0620 02:36:25.484949 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.484953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.485000 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:36:25.485374 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.485378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.485450 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.485540 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:36:25.485983 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.486786 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.488840 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.489448 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.489636 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.490245 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:36:25.490406 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:36:25.490489 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:25.490718 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 02:36:25.490808 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:36:25.491163 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.484631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34075 (local address 127.0.0.1:48994)
0620 02:36:25.484922 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:25.484944 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:25.485111 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:25.485585 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:25.485604 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:25.485618 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:36:25.485808 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.485819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.486922 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.486928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.488669 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.488686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.489643 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.489651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.489881 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.489972 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:36:25.490513 (+ 541us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:36:25.490536 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:36:25.490633 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 02:36:25.490809 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 02:36:25.495786 8437 raft_consensus.cc:1261] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Refusing update from remote peer 8ce71c935e304e8c821000dc57f05a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:25.495766 8162 raft_consensus.cc:1261] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Refusing update from remote peer 8ce71c935e304e8c821000dc57f05a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:25.496471 8556 consensus_queue.cc:1035] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:25.496848 8522 consensus_queue.cc:1035] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:25.504598 8519 tablet_replica.cc:439] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599: TxnStatusTablet state changed. Reason: New leader 8ce71c935e304e8c821000dc57f05a23. Latest consensus state: current_term: 1 leader_uuid: "8ce71c935e304e8c821000dc57f05a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } }
I20260620 02:36:25.504985 8519 tablet_replica.cc:442] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:36:25.511083 8517 tablet_replica.cc:439] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368: TxnStatusTablet state changed. Reason: New leader 8ce71c935e304e8c821000dc57f05a23. Latest consensus state: current_term: 1 leader_uuid: "8ce71c935e304e8c821000dc57f05a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } }
I20260620 02:36:25.511389 8517 tablet_replica.cc:442] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:36:25.516386 8556 tablet_replica.cc:439] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ce71c935e304e8c821000dc57f05a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } }
I20260620 02:36:25.517304 8556 tablet_replica.cc:442] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:36:25.526551 8519 tablet_replica.cc:439] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ce71c935e304e8c821000dc57f05a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } }
I20260620 02:36:25.526957 8519 tablet_replica.cc:442] T db2e82804bb54a0fb8214814af88cdf9 P 4b8e0d990ec84fb7be8fc8e9b78c0599: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:36:25.528376 8556 tablet_replica.cc:439] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ce71c935e304e8c821000dc57f05a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } }
I20260620 02:36:25.528887 8556 tablet_replica.cc:442] T db2e82804bb54a0fb8214814af88cdf9 P 8ce71c935e304e8c821000dc57f05a23: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:36:25.537571 8517 tablet_replica.cc:439] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ce71c935e304e8c821000dc57f05a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } } }
I20260620 02:36:25.537901 8517 tablet_replica.cc:442] T db2e82804bb54a0fb8214814af88cdf9 P 8b0fb1fb2a00415d8602e7b8ed8f8368: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:36:25.543581 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.528837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.0.97.1:35629)
0620 02:36:25.536615 (+ 7778us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.536622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.536656 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:36:25.536787 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.536794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.536927 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.537855 (+ 928us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:36:25.538085 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.539414 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.541104 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.541567 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.541722 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.542249 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:36:25.542852 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:36:25.543099 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:25.543150 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 02:36:25.543236 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7608,"thread_start_us":7439,"threads_started":1}
I20260620 02:36:25.566089 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.556324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45375 (local address 127.0.97.2:34075)
0620 02:36:25.556572 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.556582 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.561184 (+ 4602us) server_negotiation.cc:407] Connection header received
0620 02:36:25.561443 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.561447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.561538 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.561650 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:36:25.562038 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.562826 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.564232 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.564684 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.564832 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.565273 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:36:25.565407 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:36:25.565517 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:25.565668 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 02:36:25.565757 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:36:25.566107 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.556322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34075 (local address 127.0.97.1:45375)
0620 02:36:25.561042 (+ 4720us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:25.561063 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:25.561287 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:25.561672 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:25.561677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:25.561690 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:36:25.561900 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.561909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.562977 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.562983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.564072 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.564084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.564856 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.564861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.565010 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.565074 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:36:25.565516 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:36:25.565529 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:36:25.565632 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 02:36:25.565784 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4532,"thread_start_us":4439,"threads_started":1}
I20260620 02:36:25.577476 8437 raft_consensus.cc:1261] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Refusing update from remote peer 4b8e0d990ec84fb7be8fc8e9b78c0599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:25.577884 8297 raft_consensus.cc:1261] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Refusing update from remote peer 4b8e0d990ec84fb7be8fc8e9b78c0599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:25.578440 8577 consensus_queue.cc:1035] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:25.578924 8519 consensus_queue.cc:1035] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:25.599347 8563 mvcc.cc:205] Tried to move back new op lower bound from 7298756548918128640 to 7298756546847051776. Current Snapshot: MvccSnapshot[applied={T|T < 7298756548918128640}]
I20260620 02:36:25.626869 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.618944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55499 (local address 127.0.97.1:35629)
0620 02:36:25.619134 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.619141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.620010 (+ 869us) server_negotiation.cc:407] Connection header received
0620 02:36:25.620262 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.620269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.620357 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.620471 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:36:25.621444 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.622609 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.624793 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.625383 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.625539 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.625999 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:36:25.626159 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:36:25.626243 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:25.626418 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 02:36:25.626498 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:36:25.627310 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.618941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35629 (local address 127.0.97.2:55499)
0620 02:36:25.619894 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:25.619914 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:25.620109 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:25.620660 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:25.620665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:25.620678 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:36:25.621138 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.621250 (+ 112us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.622766 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.622772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.624620 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.624637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.625591 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.625599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.625753 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.625816 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:36:25.626293 (+ 477us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:36:25.626307 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 02:36:25.626397 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 02:36:25.626544 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":483,"threads_started":1}
I20260620 02:36:25.666878 8000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:25.667167 8000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:25.670745 8000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18: Sending DeleteTablet for 3 replicas of tablet 8e5a0462a0624ef9bd702a1ec3f3eb05
I20260620 02:36:25.671862 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5a0462a0624ef9bd702a1ec3f3eb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48976
I20260620 02:36:25.671975 8417 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5a0462a0624ef9bd702a1ec3f3eb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39028
I20260620 02:36:25.672430 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5a0462a0624ef9bd702a1ec3f3eb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46192
I20260620 02:36:25.699939 8588 tablet_replica.cc:330] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368: stopping tablet replica
I20260620 02:36:25.700229 8588 raft_consensus.cc:2229] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:25.700515 8588 raft_consensus.cc:2258] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:25.701628 8589 tablet_replica.cc:330] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599: stopping tablet replica
I20260620 02:36:25.701828 8589 raft_consensus.cc:2229] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:36:25.702066 8588 ts_tablet_manager.cc:1905] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:25.702384 8587 tablet_replica.cc:330] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23: stopping tablet replica
I20260620 02:36:25.702474 8589 raft_consensus.cc:2258] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:25.702749 8587 raft_consensus.cc:2229] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:25.703061 8587 raft_consensus.cc:2258] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:25.703567 8589 ts_tablet_manager.cc:1905] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:25.704412 8587 ts_tablet_manager.cc:1905] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:25.710398 8588 ts_tablet_manager.cc:1918] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:36:25.710637 8588 log.cc:1196] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/8e5a0462a0624ef9bd702a1ec3f3eb05
I20260620 02:36:25.711615 8588 ts_tablet_manager.cc:1939] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting consensus metadata
I20260620 02:36:25.713011 7986 catalog_manager.cc:4841] TS 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141): tablet 8e5a0462a0624ef9bd702a1ec3f3eb05 (table test-table [id=ebbdaea9736240e688090570ef44055c]) successfully deleted
I20260620 02:36:25.715993 8589 ts_tablet_manager.cc:1918] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:36:25.716158 8589 log.cc:1196] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/8e5a0462a0624ef9bd702a1ec3f3eb05
I20260620 02:36:25.716648 8589 ts_tablet_manager.cc:1939] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting consensus metadata
I20260620 02:36:25.717941 7986 catalog_manager.cc:4841] TS 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629): tablet 8e5a0462a0624ef9bd702a1ec3f3eb05 (table test-table [id=ebbdaea9736240e688090570ef44055c]) successfully deleted
I20260620 02:36:25.717968 8587 ts_tablet_manager.cc:1918] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:36:25.718216 8587 log.cc:1196] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/8e5a0462a0624ef9bd702a1ec3f3eb05
I20260620 02:36:25.719000 8587 ts_tablet_manager.cc:1939] T 8e5a0462a0624ef9bd702a1ec3f3eb05 P 8ce71c935e304e8c821000dc57f05a23: Deleting consensus metadata
I20260620 02:36:25.720604 7987 catalog_manager.cc:4841] TS 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2:34075): tablet 8e5a0462a0624ef9bd702a1ec3f3eb05 (table test-table [id=ebbdaea9736240e688090570ef44055c]) successfully deleted
Jun 20 02:36:25 dist-test-slave-rpg6 krb5kdc[7955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:36:25 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:25 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:36:25.779693 8597 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:25.758389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.0.97.62:38467)
0620 02:36:25.766005 (+ 7616us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.766012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.766036 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:36:25.766108 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.766113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.766206 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.766351 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:25.768401 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.769654 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.771761 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.772469 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.772647 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.779336 (+ 6689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43394: BlockingRecv error: recv got EOF from 127.0.0.1:43394 (error 108)
Metrics: {"server-negotiator.queue_time_us":7472,"thread_start_us":7328,"threads_started":1}
Jun 20 02:36:25 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:36:25.807720 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.785869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.0.97.62:38467)
0620 02:36:25.786074 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.786081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.786685 (+ 604us) server_negotiation.cc:407] Connection header received
0620 02:36:25.786909 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.786915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.787005 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.787132 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:25.788962 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.790270 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.792596 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.793289 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.793446 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.798978 (+ 5532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:25.798988 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:25.798994 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:25.799090 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:25.803382 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:25.804441 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:25.804449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:25.804453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:25.804640 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:25.805508 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:25.805514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:25.805517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:25.806057 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:25.806344 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:25.807058 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 02:36:25.807341 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:36:25.814281 8000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:36:25.815066 8000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:36:25.825927 8136 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8af613852b41eb8f8799d29f1ed8a8 (DEFAULT_TABLE table=test-table [id=5f4f5a8aa4bb4d18adf868ada7eebd25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:25.826472 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8af613852b41eb8f8799d29f1ed8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:25.828320 8417 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8af613852b41eb8f8799d29f1ed8a8 (DEFAULT_TABLE table=test-table [id=5f4f5a8aa4bb4d18adf868ada7eebd25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:25.828796 8417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8af613852b41eb8f8799d29f1ed8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:25.832309 8276 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8af613852b41eb8f8799d29f1ed8a8 (DEFAULT_TABLE table=test-table [id=5f4f5a8aa4bb4d18adf868ada7eebd25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:25.832834 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8af613852b41eb8f8799d29f1ed8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:25.868564 8603 tablet_bootstrap.cc:492] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Bootstrap starting.
I20260620 02:36:25.871076 8603 tablet_bootstrap.cc:654] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:25.871929 8605 tablet_bootstrap.cc:492] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Bootstrap starting.
I20260620 02:36:25.873318 8604 tablet_bootstrap.cc:492] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23: Bootstrap starting.
I20260620 02:36:25.874221 8605 tablet_bootstrap.cc:654] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:25.875308 8604 tablet_bootstrap.cc:654] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:25.876423 8605 tablet_bootstrap.cc:492] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368: No bootstrap required, opened a new log
I20260620 02:36:25.876641 8605 ts_tablet_manager.cc:1397] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 02:36:25.877580 8605 raft_consensus.cc:348] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.877995 8605 raft_consensus.cc:374] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:25.878149 8605 raft_consensus.cc:729] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fb1fb2a00415d8602e7b8ed8f8368, State: Initialized, Role: FOLLOWER
I20260620 02:36:25.878537 8605 consensus_queue.cc:260] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.879173 8603 tablet_bootstrap.cc:492] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599: No bootstrap required, opened a new log
I20260620 02:36:25.879446 8603 ts_tablet_manager.cc:1397] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260620 02:36:25.880465 8603 raft_consensus.cc:348] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.880820 8603 raft_consensus.cc:374] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:25.880935 8603 raft_consensus.cc:729] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8e0d990ec84fb7be8fc8e9b78c0599, State: Initialized, Role: FOLLOWER
I20260620 02:36:25.882104 8603 consensus_queue.cc:260] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.882315 8605 ts_tablet_manager.cc:1428] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260620 02:36:25.883584 8603 ts_tablet_manager.cc:1428] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 02:36:25.884727 8604 tablet_bootstrap.cc:492] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23: No bootstrap required, opened a new log
I20260620 02:36:25.884900 8604 ts_tablet_manager.cc:1397] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20260620 02:36:25.885844 8604 raft_consensus.cc:348] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.886628 8604 raft_consensus.cc:374] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:25.886740 8604 raft_consensus.cc:729] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce71c935e304e8c821000dc57f05a23, State: Initialized, Role: FOLLOWER
I20260620 02:36:25.887091 8604 consensus_queue.cc:260] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.888768 8604 ts_tablet_manager.cc:1428] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260620 02:36:25.899581 8517 raft_consensus.cc:482] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:36:25.899801 8517 raft_consensus.cc:504] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.901710 8517 leader_election.cc:288] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629), 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2:34075)
I20260620 02:36:25.924001 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.903278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40565 (local address 127.0.97.2:34075)
0620 02:36:25.903520 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.903528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.915976 (+ 12448us) server_negotiation.cc:407] Connection header received
0620 02:36:25.916358 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.916363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.916479 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.916554 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:25.917450 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.918553 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.921261 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.922626 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.922750 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.922881 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:25.923484 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 02:36:25.923650 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:36:25.924036 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.902105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36869 (local address 127.0.97.1:35629)
0620 02:36:25.902334 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.902344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.915504 (+ 13160us) server_negotiation.cc:407] Connection header received
0620 02:36:25.916060 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.916067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.916151 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.916249 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:25.917041 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.918346 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.921473 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.922842 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.922945 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.923047 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:25.923597 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 02:36:25.923739 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:36:25.925019 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.902812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34075 (local address 127.0.97.3:40565)
0620 02:36:25.915865 (+ 13053us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:25.915882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:25.916199 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:25.916777 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:25.916783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:25.916814 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:25.917251 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.917263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.918747 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.918752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.921053 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.921071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.923054 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.923063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.923281 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.923316 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:36:25.924428 (+ 1112us) client_negotiation.cc:240] Negotiation successful
0620 02:36:25.924582 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12914,"mutex_wait_us":10,"thread_start_us":12722,"threads_started":1}
I20260620 02:36:25.925264 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.901989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35629 (local address 127.0.97.3:36869)
0620 02:36:25.915342 (+ 13353us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:25.915375 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:25.915875 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:25.916489 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:25.916495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:25.916520 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:25.916824 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.916838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.918559 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.918564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.921053 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:25.921071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.923222 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.923240 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:25.923427 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.923461 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:36:25.924610 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0620 02:36:25.924745 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13132,"mutex_wait_us":81,"thread_start_us":13992,"threads_started":1}
I20260620 02:36:25.925932 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8af613852b41eb8f8799d29f1ed8a8" candidate_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce71c935e304e8c821000dc57f05a23" is_pre_election: true
I20260620 02:36:25.925940 8162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8af613852b41eb8f8799d29f1ed8a8" candidate_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" is_pre_election: true
I20260620 02:36:25.926210 8297 raft_consensus.cc:2454] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b0fb1fb2a00415d8602e7b8ed8f8368 in term 0.
I20260620 02:36:25.926210 8162 raft_consensus.cc:2454] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b0fb1fb2a00415d8602e7b8ed8f8368 in term 0.
I20260620 02:36:25.926859 8369 leader_election.cc:302] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b8e0d990ec84fb7be8fc8e9b78c0599, 8b0fb1fb2a00415d8602e7b8ed8f8368; no voters:
I20260620 02:36:25.927600 8517 raft_consensus.cc:2781] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:36:25.927748 8517 raft_consensus.cc:482] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:36:25.927884 8517 raft_consensus.cc:3037] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:25.930608 8517 raft_consensus.cc:504] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.931365 8517 leader_election.cc:288] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [CANDIDATE]: Term 1 election: Requested vote from peers 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629), 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2:34075)
I20260620 02:36:25.932276 8162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8af613852b41eb8f8799d29f1ed8a8" candidate_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599"
I20260620 02:36:25.932539 8162 raft_consensus.cc:3037] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:25.932950 8297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8af613852b41eb8f8799d29f1ed8a8" candidate_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce71c935e304e8c821000dc57f05a23"
I20260620 02:36:25.933192 8297 raft_consensus.cc:3037] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:25.935907 8162 raft_consensus.cc:2454] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0fb1fb2a00415d8602e7b8ed8f8368 in term 1.
I20260620 02:36:25.936414 8297 raft_consensus.cc:2454] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b0fb1fb2a00415d8602e7b8ed8f8368 in term 1.
I20260620 02:36:25.936539 8369 leader_election.cc:302] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b8e0d990ec84fb7be8fc8e9b78c0599, 8b0fb1fb2a00415d8602e7b8ed8f8368; no voters:
I20260620 02:36:25.936935 8517 raft_consensus.cc:2781] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:25.940212 8517 raft_consensus.cc:686] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 LEADER]: Becoming Leader. State: Replica: 8b0fb1fb2a00415d8602e7b8ed8f8368, State: Running, Role: LEADER
I20260620 02:36:25.940722 8517 consensus_queue.cc:237] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:25.944649 8000 catalog_manager.cc:5495] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } health_report { overall_health: HEALTHY } } }
I20260620 02:36:25.966116 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:25.954501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.0.97.3:45141)
0620 02:36:25.960504 (+ 6003us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:25.960509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:25.960532 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:36:25.960640 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:25.960643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:25.960723 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:25.960931 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:36:25.961420 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.962234 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.963616 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:25.964140 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:25.964283 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:25.964765 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:36:25.965370 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:36:25.965535 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:25.965619 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 02:36:25.965715 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5884,"thread_start_us":5712,"threads_started":1}
I20260620 02:36:25.969880 8297 raft_consensus.cc:1261] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Refusing update from remote peer 8b0fb1fb2a00415d8602e7b8ed8f8368: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:25.969880 8162 raft_consensus.cc:1261] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Refusing update from remote peer 8b0fb1fb2a00415d8602e7b8ed8f8368: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:25.970902 8517 consensus_queue.cc:1035] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:25.971624 8615 consensus_queue.cc:1035] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:25.998932 7999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:25.999224 7999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:26.003481 7999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18: Sending DeleteTablet for 3 replicas of tablet 2c8af613852b41eb8f8799d29f1ed8a8
I20260620 02:36:26.004477 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8af613852b41eb8f8799d29f1ed8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46192
I20260620 02:36:26.004499 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8af613852b41eb8f8799d29f1ed8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48976
I20260620 02:36:26.004827 8589 tablet_replica.cc:330] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599: stopping tablet replica
I20260620 02:36:26.004855 8587 tablet_replica.cc:330] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23: stopping tablet replica
I20260620 02:36:26.005052 8587 raft_consensus.cc:2229] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:26.005295 8587 raft_consensus.cc:2258] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:26.005537 8589 raft_consensus.cc:2229] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:26.005815 8589 raft_consensus.cc:2258] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:26.006959 8587 ts_tablet_manager.cc:1905] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:26.008549 8589 ts_tablet_manager.cc:1905] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:26.012174 8417 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8af613852b41eb8f8799d29f1ed8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39028
I20260620 02:36:26.012519 8588 tablet_replica.cc:330] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368: stopping tablet replica
I20260620 02:36:26.012754 8588 raft_consensus.cc:2229] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:36:26.013326 8588 raft_consensus.cc:2258] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:26.015084 7999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 02:36:26.015789 8588 ts_tablet_manager.cc:1905] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 02:36:26.015925 7999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:36:26.017371 8587 ts_tablet_manager.cc:1918] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:26.017546 8587 log.cc:1196] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/2c8af613852b41eb8f8799d29f1ed8a8
I20260620 02:36:26.017977 8587 ts_tablet_manager.cc:1939] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8ce71c935e304e8c821000dc57f05a23: Deleting consensus metadata
I20260620 02:36:26.019330 7987 catalog_manager.cc:4841] TS 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2:34075): tablet 2c8af613852b41eb8f8799d29f1ed8a8 (table test-table [id=5f4f5a8aa4bb4d18adf868ada7eebd25]) successfully deleted
I20260620 02:36:26.020409 8589 ts_tablet_manager.cc:1918] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:26.020637 8589 log.cc:1196] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/2c8af613852b41eb8f8799d29f1ed8a8
I20260620 02:36:26.021078 8589 ts_tablet_manager.cc:1939] T 2c8af613852b41eb8f8799d29f1ed8a8 P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting consensus metadata
I20260620 02:36:26.022855 7986 catalog_manager.cc:4841] TS 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629): tablet 2c8af613852b41eb8f8799d29f1ed8a8 (table test-table [id=5f4f5a8aa4bb4d18adf868ada7eebd25]) successfully deleted
I20260620 02:36:26.023048 8588 ts_tablet_manager.cc:1918] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:26.023213 8588 log.cc:1196] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/2c8af613852b41eb8f8799d29f1ed8a8
I20260620 02:36:26.023665 8588 ts_tablet_manager.cc:1939] T 2c8af613852b41eb8f8799d29f1ed8a8 P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting consensus metadata
I20260620 02:36:26.024850 8136 tablet_service.cc:1467] Processing CreateTablet for tablet a73e3edaa2874404bc9e23eb5367175f (DEFAULT_TABLE table=test-table [id=f3143a685638482db7cffd59140c8251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:26.025110 8276 tablet_service.cc:1467] Processing CreateTablet for tablet a73e3edaa2874404bc9e23eb5367175f (DEFAULT_TABLE table=test-table [id=f3143a685638482db7cffd59140c8251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:26.025357 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73e3edaa2874404bc9e23eb5367175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:26.025578 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73e3edaa2874404bc9e23eb5367175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:26.031275 8604 tablet_bootstrap.cc:492] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23: Bootstrap starting.
I20260620 02:36:26.032218 7986 catalog_manager.cc:4841] TS 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141): tablet 2c8af613852b41eb8f8799d29f1ed8a8 (table test-table [id=5f4f5a8aa4bb4d18adf868ada7eebd25]) successfully deleted
I20260620 02:36:26.033118 8604 tablet_bootstrap.cc:654] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:26.033087 8417 tablet_service.cc:1467] Processing CreateTablet for tablet a73e3edaa2874404bc9e23eb5367175f (DEFAULT_TABLE table=test-table [id=f3143a685638482db7cffd59140c8251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:26.033607 8417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a73e3edaa2874404bc9e23eb5367175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:26.044937 8603 tablet_bootstrap.cc:492] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599: Bootstrap starting.
I20260620 02:36:26.045225 8604 tablet_bootstrap.cc:492] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23: No bootstrap required, opened a new log
I20260620 02:36:26.045418 8604 ts_tablet_manager.cc:1397] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260620 02:36:26.047132 8603 tablet_bootstrap.cc:654] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:26.046365 8604 raft_consensus.cc:348] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:26.048185 8604 raft_consensus.cc:374] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:26.048287 8604 raft_consensus.cc:729] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce71c935e304e8c821000dc57f05a23, State: Initialized, Role: FOLLOWER
I20260620 02:36:26.048589 8604 consensus_queue.cc:260] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:26.049379 8604 ts_tablet_manager.cc:1428] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260620 02:36:26.052282 8605 tablet_bootstrap.cc:492] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368: Bootstrap starting.
I20260620 02:36:26.054268 8605 tablet_bootstrap.cc:654] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:26.059204 8605 tablet_bootstrap.cc:492] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368: No bootstrap required, opened a new log
I20260620 02:36:26.059415 8605 ts_tablet_manager.cc:1397] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260620 02:36:26.060384 8605 raft_consensus.cc:348] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:26.060719 8603 tablet_bootstrap.cc:492] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599: No bootstrap required, opened a new log
I20260620 02:36:26.060848 8605 raft_consensus.cc:374] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:26.060974 8605 raft_consensus.cc:729] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0fb1fb2a00415d8602e7b8ed8f8368, State: Initialized, Role: FOLLOWER
I20260620 02:36:26.061136 8603 ts_tablet_manager.cc:1397] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.004s
I20260620 02:36:26.061326 8605 consensus_queue.cc:260] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:26.061447 8567 raft_consensus.cc:482] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:36:26.061627 8567 raft_consensus.cc:504] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:26.062254 8603 raft_consensus.cc:348] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:26.062443 8567 leader_election.cc:288] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629), 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141)
I20260620 02:36:26.062639 8603 raft_consensus.cc:374] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:26.062783 8603 raft_consensus.cc:729] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8e0d990ec84fb7be8fc8e9b78c0599, State: Initialized, Role: FOLLOWER
I20260620 02:36:26.062836 8605 ts_tablet_manager.cc:1428] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 02:36:26.064092 8162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e3edaa2874404bc9e23eb5367175f" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" is_pre_election: true
I20260620 02:36:26.064388 8603 consensus_queue.cc:260] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:26.064658 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e3edaa2874404bc9e23eb5367175f" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" is_pre_election: true
I20260620 02:36:26.064958 8437 raft_consensus.cc:2454] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 0.
I20260620 02:36:26.065156 8162 raft_consensus.cc:2454] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 0.
I20260620 02:36:26.065565 8229 leader_election.cc:302] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b0fb1fb2a00415d8602e7b8ed8f8368, 8ce71c935e304e8c821000dc57f05a23; no voters:
I20260620 02:36:26.067490 8603 ts_tablet_manager.cc:1428] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260620 02:36:26.068167 8567 raft_consensus.cc:2781] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:36:26.068302 8567 raft_consensus.cc:482] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:36:26.068392 8567 raft_consensus.cc:3037] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:26.071653 8567 raft_consensus.cc:504] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:26.073990 8162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e3edaa2874404bc9e23eb5367175f" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599"
I20260620 02:36:26.074266 8162 raft_consensus.cc:3037] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:26.076130 8567 leader_election.cc:288] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 election: Requested vote from peers 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629), 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141)
I20260620 02:36:26.077054 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73e3edaa2874404bc9e23eb5367175f" candidate_uuid: "8ce71c935e304e8c821000dc57f05a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368"
I20260620 02:36:26.077320 8437 raft_consensus.cc:3037] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:26.077750 8162 raft_consensus.cc:2454] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 1.
I20260620 02:36:26.078271 8229 leader_election.cc:302] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b8e0d990ec84fb7be8fc8e9b78c0599, 8ce71c935e304e8c821000dc57f05a23; no voters:
I20260620 02:36:26.078686 8567 raft_consensus.cc:2781] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:26.079370 8567 raft_consensus.cc:686] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 1 LEADER]: Becoming Leader. State: Replica: 8ce71c935e304e8c821000dc57f05a23, State: Running, Role: LEADER
I20260620 02:36:26.079685 8567 consensus_queue.cc:237] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } }
I20260620 02:36:26.082599 8437 raft_consensus.cc:2454] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce71c935e304e8c821000dc57f05a23 in term 1.
I20260620 02:36:26.086356 7999 catalog_manager.cc:5495] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8ce71c935e304e8c821000dc57f05a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce71c935e304e8c821000dc57f05a23" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20260620 02:36:26.116125 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:26.109096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49010 (local address 127.0.97.2:34075)
0620 02:36:26.109275 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:26.109281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:26.109357 (+ 76us) server_negotiation.cc:407] Connection header received
0620 02:36:26.109568 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:26.109573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:26.109674 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:26.109783 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:36:26.110233 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.111381 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.113453 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.114101 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.114259 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:26.114777 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:36:26.114941 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:36:26.115043 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:26.115643 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 02:36:26.115733 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:36:26.133553 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:26.127368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35767 (local address 127.0.97.2:34075)
0620 02:36:26.127526 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:26.127532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:26.127549 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:36:26.127614 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:26.127620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:26.127693 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:26.127780 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:36:26.128283 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.129365 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.131355 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.132092 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.132229 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:26.132688 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:36:26.132835 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:36:26.132926 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:26.133102 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 02:36:26.133184 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 02:36:26.133556 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:26.126788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34075 (local address 127.0.97.2:35767)
0620 02:36:26.127047 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:26.127067 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:26.127187 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:26.127889 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:26.127894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:26.127910 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:36:26.128124 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:26.128135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.129521 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.129528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:26.131179 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:26.131195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.132265 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.132271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:26.132457 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:26.132510 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:36:26.132954 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:36:26.132966 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 02:36:26.133061 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:36:26.133218 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 02:36:26.145489 8437 raft_consensus.cc:1261] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Refusing update from remote peer 8ce71c935e304e8c821000dc57f05a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:26.146035 8162 raft_consensus.cc:1261] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Refusing update from remote peer 8ce71c935e304e8c821000dc57f05a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:26.146117 8567 consensus_queue.cc:1035] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b0fb1fb2a00415d8602e7b8ed8f8368" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:26.147181 8567 consensus_queue.cc:1035] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b8e0d990ec84fb7be8fc8e9b78c0599" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:26.152643 8561 mvcc.cc:205] Tried to move back new op lower bound from 7298756551245287424 to 7298756550986018816. Current Snapshot: MvccSnapshot[applied={T|T < 7298756551245287424}]
I20260620 02:36:26.241338 7999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:26.241657 7999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:26.245688 7999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73dee90f60c446cfab44b429aeac1f18: Sending DeleteTablet for 3 replicas of tablet a73e3edaa2874404bc9e23eb5367175f
I20260620 02:36:26.246682 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet a73e3edaa2874404bc9e23eb5367175f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:26 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46192
I20260620 02:36:26.247035 8589 tablet_replica.cc:330] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599: stopping tablet replica
I20260620 02:36:26.247228 8589 raft_consensus.cc:2229] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:26.247236 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet a73e3edaa2874404bc9e23eb5367175f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:26 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48976
I20260620 02:36:26.247422 8589 raft_consensus.cc:2258] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:26.247520 8587 tablet_replica.cc:330] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23: stopping tablet replica
I20260620 02:36:26.247704 8587 raft_consensus.cc:2229] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:36:26.248137 8587 raft_consensus.cc:2258] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:26.248865 8589 ts_tablet_manager.cc:1905] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:26.249045 8417 tablet_service.cc:1514] Processing DeleteTablet for tablet a73e3edaa2874404bc9e23eb5367175f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:26 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39028
I20260620 02:36:26.249351 8588 tablet_replica.cc:330] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368: stopping tablet replica
I20260620 02:36:26.249526 8588 raft_consensus.cc:2229] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:26.249727 8588 raft_consensus.cc:2258] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:26.250612 8587 ts_tablet_manager.cc:1905] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:26.253115 8588 ts_tablet_manager.cc:1905] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:26.257406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 8071
I20260620 02:36:26.261721 8588 ts_tablet_manager.cc:1918] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:36:26.261891 8588 log.cc:1196] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/a73e3edaa2874404bc9e23eb5367175f
I20260620 02:36:26.262394 8588 ts_tablet_manager.cc:1939] T a73e3edaa2874404bc9e23eb5367175f P 8b0fb1fb2a00415d8602e7b8ed8f8368: Deleting consensus metadata
I20260620 02:36:26.263676 7986 catalog_manager.cc:4841] TS 8b0fb1fb2a00415d8602e7b8ed8f8368 (127.0.97.3:45141): tablet a73e3edaa2874404bc9e23eb5367175f (table test-table [id=f3143a685638482db7cffd59140c8251]) successfully deleted
I20260620 02:36:26.266108 8587 ts_tablet_manager.cc:1918] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:36:26.268332 8587 log.cc:1196] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/a73e3edaa2874404bc9e23eb5367175f
I20260620 02:36:26.268759 8589 ts_tablet_manager.cc:1918] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:36:26.268906 8589 log.cc:1196] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/a73e3edaa2874404bc9e23eb5367175f
I20260620 02:36:26.269315 8589 ts_tablet_manager.cc:1939] T a73e3edaa2874404bc9e23eb5367175f P 4b8e0d990ec84fb7be8fc8e9b78c0599: Deleting consensus metadata
I20260620 02:36:26.269785 8587 ts_tablet_manager.cc:1939] T a73e3edaa2874404bc9e23eb5367175f P 8ce71c935e304e8c821000dc57f05a23: Deleting consensus metadata
I20260620 02:36:26.270527 7986 catalog_manager.cc:4841] TS 4b8e0d990ec84fb7be8fc8e9b78c0599 (127.0.97.1:35629): tablet a73e3edaa2874404bc9e23eb5367175f (table test-table [id=f3143a685638482db7cffd59140c8251]) successfully deleted
I20260620 02:36:26.271941 7987 catalog_manager.cc:4841] TS 8ce71c935e304e8c821000dc57f05a23 (127.0.97.2:34075): tablet a73e3edaa2874404bc9e23eb5367175f (table test-table [id=f3143a685638482db7cffd59140c8251]) successfully deleted
Jun 20 02:36:26 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 02:36:26.306490 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:26.260055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.0.97.1:35629)
0620 02:36:26.260242 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:26.260250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:26.275206 (+ 14956us) server_negotiation.cc:407] Connection header received
0620 02:36:26.275697 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:26.275704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:26.275807 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:26.275909 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:26.277899 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.279105 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.281035 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.281737 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.281865 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:26.294731 (+ 12866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:26.294750 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:26.294772 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:26.294885 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:26.302186 (+ 7301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:26.303314 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:26.303325 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:26.303328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:26.303541 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:26.304145 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:26.304152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:26.304156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:26.304681 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:26.304968 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:26.305700 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 02:36:26.305975 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:36:26.308599 8202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:26.479099 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 8071
I20260620 02:36:26.504535 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 8211
Jun 20 02:36:26 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 02:36:26.523698 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:26.505774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.0.97.2:34075)
0620 02:36:26.506033 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:26.506040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:26.506067 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:36:26.506140 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:26.506146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:26.506238 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:26.506379 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:26.508044 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.509210 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.510561 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.511106 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.511267 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:26.517215 (+ 5948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:26.517234 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:26.517251 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:26.517297 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:26.520573 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:26.521419 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:26.521436 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:26.521439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:26.521573 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:26.521958 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:26.521965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:26.521968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:26.522356 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:26.522730 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:26.523108 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 02:36:26.523346 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 02:36:26.524379 8342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:26.672945 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 8211
W20260620 02:36:26.700402 8368 connection.cc:535] server connection from 127.0.97.2:53397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:36:26.700856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 8351
Jun 20 02:36:26 dist-test-slave-rpg6 krb5kdc[7955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 02:36:26.720561 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:26.701802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.0.97.3:45141)
0620 02:36:26.702299 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:26.702303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:26.702320 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:36:26.702373 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:26.702376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:26.702441 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:26.702535 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:26.704144 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.705127 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.707459 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:26.708156 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:26.708354 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:26.713447 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:26.713463 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:26.713703 (+ 240us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:26.713802 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:26.717224 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:26.717999 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:26.718007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:26.718009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:26.718207 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:26.718841 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:26.718845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:26.718848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:26.719265 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:26.719505 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:26.719924 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 02:36:26.720110 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":262,"threads_started":1}
I20260620 02:36:26.721324 8482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:26.865051 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 8351
I20260620 02:36:26.889199 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 7968
I20260620 02:36:26.890538 8040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:27.006948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 7968
2026-06-20T02:36:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6310 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:36:27 dist-test-slave-rpg6 krb5kdc[8637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:36:27 dist-test-slave-rpg6 krb5kdc[8637](info): set up 2 sockets
Jun 20 02:36:27 dist-test-slave-rpg6 krb5kdc[8637](info): commencing operation
krb5kdc: starting...
W20260620 02:36:28.157013 388 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:36:28 dist-test-slave-rpg6 krb5kdc[8637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:36:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:36:28.417152 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42705
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:28.772874 8650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:28.773211 8650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:28.773303 8650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:28.780987 8650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:28.781128 8650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:28.781176 8650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:28.781226 8650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:28.781276 8650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:36:28.791775 8650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:28.793131 8650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:28.794404 8650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:28.808266 8658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:28.808460 8655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:28.808434 8656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:28.809724 8650 server_base.cc:1034] running on GCE node
I20260620 02:36:28.810607 8650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:28.812453 8650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:28.813700 8650 hybrid_clock.cc:648] HybridClock initialized: now 1781922988813640 us; error 80 us; skew 500 ppm
I20260620 02:36:28.814131 8650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T02:36:28Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1880 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:36:28 dist-test-slave-rpg6 krb5kdc[8664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:36:28 dist-test-slave-rpg6 krb5kdc[8664](info): set up 2 sockets
Jun 20 02:36:28 dist-test-slave-rpg6 krb5kdc[8664](info): commencing operation
krb5kdc: starting...
W20260620 02:36:30.017570 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:36:30 dist-test-slave-rpg6 krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:36:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:36:30.276589 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45503
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:30.636593 8677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:30.636929 8677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:30.637092 8677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:30.645704 8677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:30.645850 8677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:30.645943 8677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:30.646034 8677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:30.646114 8677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:36:30.656903 8677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:30.658366 8677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:30.659791 8677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:30.673138 8683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:30.673163 8682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:30.673457 8685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:30.675588 8677 server_base.cc:1034] running on GCE node
I20260620 02:36:30.676612 8677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:30.678444 8677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:30.680225 8677 hybrid_clock.cc:648] HybridClock initialized: now 1781922990680214 us; error 168 us; skew 500 ppm
I20260620 02:36:30.680713 8677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:30 dist-test-slave-rpg6 krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:30.684252 8677 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:36:30.685846 8677 webserver.cc:460] Webserver started at http://127.0.97.62:40103/ using document root <none> and password file <none>
I20260620 02:36:30.686580 8677 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:30.686702 8677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:30.686995 8677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:30.690263 8677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "0f35cdb528ef426eaeec5c00e44b827a"
format_stamp: "Formatted at 2026-06-20 02:36:30 on dist-test-slave-rpg6"
server_key: "e482e9d8771e39416debdfdef3997649"
server_key_iv: "9fd131afae6be6af5fbb70eed2795e1d"
server_key_version: "encryptionkey@0"
I20260620 02:36:30.691083 8677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "0f35cdb528ef426eaeec5c00e44b827a"
format_stamp: "Formatted at 2026-06-20 02:36:30 on dist-test-slave-rpg6"
server_key: "e482e9d8771e39416debdfdef3997649"
server_key_iv: "9fd131afae6be6af5fbb70eed2795e1d"
server_key_version: "encryptionkey@0"
I20260620 02:36:30.697662 8677 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:36:30.701926 8692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:30.704490 8677 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 02:36:30.704747 8677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "0f35cdb528ef426eaeec5c00e44b827a"
format_stamp: "Formatted at 2026-06-20 02:36:30 on dist-test-slave-rpg6"
server_key: "e482e9d8771e39416debdfdef3997649"
server_key_iv: "9fd131afae6be6af5fbb70eed2795e1d"
server_key_version: "encryptionkey@0"
I20260620 02:36:30.705014 8677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:30.723037 8677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:30.728818 8677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:30.729149 8677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:30.752339 8677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45503
I20260620 02:36:30.752385 8743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45503 every 8 connection(s)
I20260620 02:36:30.753988 8677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:30.754786 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 8677
I20260620 02:36:30.755048 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:36:30.756580 388 external_mini_cluster.cc:1428] Setting key cea8c3f25d34136b47c1f5f4d9b35c63
I20260620 02:36:30.760528 8744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:36:30 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:36:30.776679 8744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a: Bootstrap starting.
I20260620 02:36:30.781322 8744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:30.782809 8744 log.cc:824] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:30.787115 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:30.759303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.0.97.62:45503)
0620 02:36:30.760357 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:30.760369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:30.760419 (+ 50us) server_negotiation.cc:407] Connection header received
0620 02:36:30.762141 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:30.762164 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:30.762612 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:30.762974 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:30.765167 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:30.766802 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:30.768887 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:30.769702 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:30.769877 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:30.776214 (+ 6337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:30.776227 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:30.776252 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:30.776319 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:30.780810 (+ 4491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:30.781897 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:30.781908 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:30.781918 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:30.782144 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:30.782554 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:30.782561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:30.782570 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:30.783686 (+ 1116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:30.784105 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:30.784966 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 02:36:30.785578 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":592,"threads_started":1}
I20260620 02:36:30.787294 8744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a: No bootstrap required, opened a new log
I20260620 02:36:30.796379 8744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } }
I20260620 02:36:30.797125 8744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:30.797255 8744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f35cdb528ef426eaeec5c00e44b827a, State: Initialized, Role: FOLLOWER
I20260620 02:36:30.798046 8744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } }
I20260620 02:36:30.798318 8744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:30.798485 8744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:30.798686 8744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:30.801192 8744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } }
I20260620 02:36:30.801843 8744 leader_election.cc:302] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f35cdb528ef426eaeec5c00e44b827a; no voters:
I20260620 02:36:30.802527 8744 leader_election.cc:288] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:36:30.802707 8749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:30.803440 8749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 1 LEADER]: Becoming Leader. State: Replica: 0f35cdb528ef426eaeec5c00e44b827a, State: Running, Role: LEADER
I20260620 02:36:30.804092 8749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } }
I20260620 02:36:30.805043 8744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:30.807453 8750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f35cdb528ef426eaeec5c00e44b827a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } } }
I20260620 02:36:30.809000 8750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:30.808823 8751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f35cdb528ef426eaeec5c00e44b827a. Latest consensus state: current_term: 1 leader_uuid: "0f35cdb528ef426eaeec5c00e44b827a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } } }
I20260620 02:36:30.809194 8751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:30.811673 8757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:30.817101 8757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:30.828064 8757 catalog_manager.cc:1347] Generated new cluster ID: 002346b9aee7401997201aee67d6ed7f
I20260620 02:36:30.828217 8757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:30.856398 8757 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:36:30.858417 8757 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:30.867424 8757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a: Generated new TSK 0
I20260620 02:36:30.868407 8757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:36:30.944063 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45503
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:31.342445 8770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:31.342780 8770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:31.342970 8770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:31.350790 8770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:31.350936 8770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:31.351145 8770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:36:31.362891 8770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:31.364957 8770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:31.366588 8770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:31.382488 8775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:31.382587 8778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:31.382566 8776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:31.383332 8770 server_base.cc:1034] running on GCE node
I20260620 02:36:31.384157 8770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:31.385305 8770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:31.386584 8770 hybrid_clock.cc:648] HybridClock initialized: now 1781922991386531 us; error 55 us; skew 500 ppm
I20260620 02:36:31.387035 8770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:31 dist-test-slave-rpg6 krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:31.390703 8770 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:36:31.392526 8770 webserver.cc:460] Webserver started at http://127.0.97.1:40997/ using document root <none> and password file <none>
I20260620 02:36:31.393232 8770 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:31.393348 8770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:31.393623 8770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:31.396876 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "d0f83e857c2c43fd92a3c657c94188c7"
format_stamp: "Formatted at 2026-06-20 02:36:31 on dist-test-slave-rpg6"
server_key: "5ce46ef7280bfc195a598acc8d3fecd7"
server_key_iv: "6e461bb28316a183729fd7bee1169ed1"
server_key_version: "encryptionkey@0"
I20260620 02:36:31.397722 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d0f83e857c2c43fd92a3c657c94188c7"
format_stamp: "Formatted at 2026-06-20 02:36:31 on dist-test-slave-rpg6"
server_key: "5ce46ef7280bfc195a598acc8d3fecd7"
server_key_iv: "6e461bb28316a183729fd7bee1169ed1"
server_key_version: "encryptionkey@0"
I20260620 02:36:31.404199 8770 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:36:31.408710 8785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:31.410534 8770 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:36:31.410817 8770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "d0f83e857c2c43fd92a3c657c94188c7"
format_stamp: "Formatted at 2026-06-20 02:36:31 on dist-test-slave-rpg6"
server_key: "5ce46ef7280bfc195a598acc8d3fecd7"
server_key_iv: "6e461bb28316a183729fd7bee1169ed1"
server_key_version: "encryptionkey@0"
I20260620 02:36:31.411096 8770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:31.435042 8770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:31.440737 8770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:31.441087 8770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:31.442139 8770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:31.443689 8770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:31.443768 8770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:31.443924 8770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:31.443986 8770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:31.479355 8770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36527
I20260620 02:36:31.479444 8897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36527 every 8 connection(s)
I20260620 02:36:31.480726 8770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:36:31.484606 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 8770
I20260620 02:36:31.484877 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:36:31.485561 388 external_mini_cluster.cc:1428] Setting key 76ce44dd0221d6337073a0e6a715c6fd
Jun 20 02:36:31 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:36:31.502498 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:31.483425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53775 (local address 127.0.97.62:45503)
0620 02:36:31.483948 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:31.483953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:31.485150 (+ 1197us) server_negotiation.cc:407] Connection header received
0620 02:36:31.487207 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:31.487212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:31.487307 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:31.487421 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:31.489021 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:31.489750 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:31.491343 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:31.491746 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:31.491903 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:31.495952 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:31.495958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:31.495961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:31.495993 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:31.498660 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:31.499730 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:31.499735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:31.499738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:31.499945 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:31.500583 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:31.500587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:31.500589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:31.500908 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:31.501028 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:31.502019 (+ 991us) server_negotiation.cc:299] Negotiation successful
0620 02:36:31.502237 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260620 02:36:31.504096 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:31.484099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:53775)
0620 02:36:31.484962 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:31.485007 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:31.486926 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:31.487662 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:31.487678 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:31.488263 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:31.488842 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:31.488865 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:31.489937 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:31.489942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:31.491193 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:31.491205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:31.492000 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:31.492009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:31.492247 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:31.492946 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:31.492972 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:31.495640 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:31.498857 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:31.498869 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:31.498892 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:31.499545 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:31.500059 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:31.500067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:31.500073 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:31.500331 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:31.501048 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:31.501060 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:31.501539 (+ 479us) client_negotiation.cc:769] Sending connection context
0620 02:36:31.501874 (+ 335us) client_negotiation.cc:240] Negotiation successful
0620 02:36:31.502507 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":358,"threads_started":1}
I20260620 02:36:31.505964 8898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45503
I20260620 02:36:31.506364 8898 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:31.507189 8898 heartbeater.cc:511] Master 127.0.97.62:45503 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:36:31.509697 8709 ts_manager.cc:194] Registered new tserver with Master: d0f83e857c2c43fd92a3c657c94188c7 (127.0.97.1:36527)
I20260620 02:36:31.511402 8709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:36:31.539885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45503
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:31.883260 8904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:31.883577 8904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:31.883744 8904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:31.891474 8904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:31.891606 8904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:31.891918 8904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:36:31.902915 8904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:31.904348 8904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:31.905758 8904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:31.921144 8909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:31.921216 8912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:31.921168 8910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:31.921929 8904 server_base.cc:1034] running on GCE node
I20260620 02:36:31.922766 8904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:31.923882 8904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:31.925123 8904 hybrid_clock.cc:648] HybridClock initialized: now 1781922991925088 us; error 39 us; skew 500 ppm
I20260620 02:36:31.925537 8904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:31 dist-test-slave-rpg6 krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:31.929090 8904 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:36:31.930714 8904 webserver.cc:460] Webserver started at http://127.0.97.2:42355/ using document root <none> and password file <none>
I20260620 02:36:31.931416 8904 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:31.931517 8904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:31.931813 8904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:31.934942 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "8fac1228e48d46a68dfd8bcb3c903c2f"
format_stamp: "Formatted at 2026-06-20 02:36:31 on dist-test-slave-rpg6"
server_key: "763f0e62f09c39b9be85478ae301d061"
server_key_iv: "a32699d20f86ef4998aaff8981c03a06"
server_key_version: "encryptionkey@0"
I20260620 02:36:31.935734 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8fac1228e48d46a68dfd8bcb3c903c2f"
format_stamp: "Formatted at 2026-06-20 02:36:31 on dist-test-slave-rpg6"
server_key: "763f0e62f09c39b9be85478ae301d061"
server_key_iv: "a32699d20f86ef4998aaff8981c03a06"
server_key_version: "encryptionkey@0"
I20260620 02:36:31.941845 8904 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 02:36:31.945679 8919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:31.947377 8904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 02:36:31.947633 8904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "8fac1228e48d46a68dfd8bcb3c903c2f"
format_stamp: "Formatted at 2026-06-20 02:36:31 on dist-test-slave-rpg6"
server_key: "763f0e62f09c39b9be85478ae301d061"
server_key_iv: "a32699d20f86ef4998aaff8981c03a06"
server_key_version: "encryptionkey@0"
I20260620 02:36:31.947903 8904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:31.965837 8904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:31.970469 8904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:31.970791 8904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:31.971848 8904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:31.973340 8904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:31.973446 8904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:31.973599 8904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:31.973672 8904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:32.009199 8904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37869
I20260620 02:36:32.009382 9031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37869 every 8 connection(s)
I20260620 02:36:32.010823 8904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:36:32.014281 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 8904
I20260620 02:36:32.014508 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:36:32.015062 388 external_mini_cluster.cc:1428] Setting key 5c152448dab6139394af6da0c92bfa4b
Jun 20 02:36:32 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:36:32.034339 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:32.014130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59799 (local address 127.0.97.62:45503)
0620 02:36:32.014793 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:32.014799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:32.015914 (+ 1115us) server_negotiation.cc:407] Connection header received
0620 02:36:32.018074 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:32.018080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:32.018189 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:32.018309 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:32.019903 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.020628 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.022253 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.022685 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.022853 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:32.026612 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:32.026620 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:32.026625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:32.026671 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:32.030450 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:32.031307 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:32.031315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:32.031322 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:32.031479 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:32.032000 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:32.032008 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:32.032014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:32.032437 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:32.032608 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:32.033528 (+ 920us) server_negotiation.cc:299] Negotiation successful
0620 02:36:32.033808 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":248,"threads_started":1}
I20260620 02:36:32.035153 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:32.014703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:59799)
0620 02:36:32.015603 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:32.015754 (+ 151us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:32.017783 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:32.018570 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:32.018579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:32.019111 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:32.019701 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:32.019719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.020811 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.020815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:32.022092 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:32.022104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.023103 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.023113 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:32.023351 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:32.024151 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:32.024175 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:32.026353 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:32.030639 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:32.030649 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:32.030666 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:32.031139 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:32.031602 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:32.031605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:32.031608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:32.031819 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:32.032638 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:32.032646 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:32.033036 (+ 390us) client_negotiation.cc:769] Sending connection context
0620 02:36:32.033382 (+ 346us) client_negotiation.cc:240] Negotiation successful
0620 02:36:32.033845 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":362,"threads_started":1}
I20260620 02:36:32.037174 9032 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45503
I20260620 02:36:32.037573 9032 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:36:32.038661 9032 heartbeater.cc:511] Master 127.0.97.62:45503 requested a full tablet report, sending...
I20260620 02:36:32.040469 8709 ts_manager.cc:194] Registered new tserver with Master: 8fac1228e48d46a68dfd8bcb3c903c2f (127.0.97.2:37869)
I20260620 02:36:32.041504 8709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:36:32.074514 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45503
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:32.438792 9038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:32.439231 9038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:32.439347 9038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:32.447357 9038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:32.447501 9038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:32.447623 9038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:36:32.458249 9038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:32.459621 9038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:32.461081 9038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:32.475983 9043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:32.475906 9046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:32.475945 9044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:32.477164 9038 server_base.cc:1034] running on GCE node
I20260620 02:36:32.477927 9038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:32.479161 9038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:32.480423 9038 hybrid_clock.cc:648] HybridClock initialized: now 1781922992480350 us; error 71 us; skew 500 ppm
I20260620 02:36:32.480852 9038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:32 dist-test-slave-rpg6 krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:32.484426 9038 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:36:32.486127 9038 webserver.cc:460] Webserver started at http://127.0.97.3:45819/ using document root <none> and password file <none>
I20260620 02:36:32.486858 9038 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:32.486970 9038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:32.487236 9038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:32.490552 9038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "f5b667fa96de4f6093058fb04159d446"
format_stamp: "Formatted at 2026-06-20 02:36:32 on dist-test-slave-rpg6"
server_key: "31b1c2b7f03447f9e35d23e785a639f0"
server_key_iv: "7e6f2a2d13b7ac5b33baca22d80a8f2a"
server_key_version: "encryptionkey@0"
I20260620 02:36:32.491396 9038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f5b667fa96de4f6093058fb04159d446"
format_stamp: "Formatted at 2026-06-20 02:36:32 on dist-test-slave-rpg6"
server_key: "31b1c2b7f03447f9e35d23e785a639f0"
server_key_iv: "7e6f2a2d13b7ac5b33baca22d80a8f2a"
server_key_version: "encryptionkey@0"
I20260620 02:36:32.497403 9038 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260620 02:36:32.501271 9053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:32.503294 9038 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:36:32.503546 9038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "f5b667fa96de4f6093058fb04159d446"
format_stamp: "Formatted at 2026-06-20 02:36:32 on dist-test-slave-rpg6"
server_key: "31b1c2b7f03447f9e35d23e785a639f0"
server_key_iv: "7e6f2a2d13b7ac5b33baca22d80a8f2a"
server_key_version: "encryptionkey@0"
I20260620 02:36:32.503759 9038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:32.515280 8898 heartbeater.cc:503] Master 127.0.97.62:45503 was elected leader, sending a full tablet report...
I20260620 02:36:32.534607 9038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:32.540180 9038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:32.540524 9038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:32.541607 9038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:32.543048 9038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:32.543169 9038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:32.543324 9038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:32.543401 9038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:32.580199 9038 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37779
I20260620 02:36:32.580307 9165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37779 every 8 connection(s)
I20260620 02:36:32.581604 9038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:36:32.591926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 9038
I20260620 02:36:32.592144 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:36:32.592684 388 external_mini_cluster.cc:1428] Setting key 1b9be89dda1e6dd3c97709cdaf8c13da
Jun 20 02:36:32 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:36:32.605657 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:32.584628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52889 (local address 127.0.97.62:45503)
0620 02:36:32.585110 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:32.585115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:32.585954 (+ 839us) server_negotiation.cc:407] Connection header received
0620 02:36:32.588062 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:32.588068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:32.588175 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:32.588295 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:32.589951 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.590764 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.593135 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.593720 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.593857 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:32.598355 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:32.598361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:32.598365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:32.598404 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:32.601196 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:32.602293 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:32.602313 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:32.602316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:32.602556 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:32.603152 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:32.603158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:32.603161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:32.603535 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:32.603666 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:32.604715 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0620 02:36:32.605205 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":262,"threads_started":1}
I20260620 02:36:32.606243 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:32.585040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:52889)
0620 02:36:32.585773 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:32.585819 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:32.587683 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:32.588572 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:32.588581 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:32.589129 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:32.589758 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:32.589776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.590952 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.590957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:32.592937 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:32.592964 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.594169 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.594181 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:32.594515 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:32.595334 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:32.595370 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:32.597978 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:32.601423 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:32.601432 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:32.601449 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:32.601994 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:32.602726 (+ 732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:32.602731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:32.602733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:32.602958 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:32.603685 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:32.603692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:32.604149 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 02:36:32.604414 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 02:36:32.604875 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":285,"threads_started":1}
I20260620 02:36:32.608717 9166 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45503
I20260620 02:36:32.609447 9166 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:32.610693 9166 heartbeater.cc:511] Master 127.0.97.62:45503 requested a full tablet report, sending...
I20260620 02:36:32.612779 8709 ts_manager.cc:194] Registered new tserver with Master: f5b667fa96de4f6093058fb04159d446 (127.0.97.3:37779)
I20260620 02:36:32.613869 8709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52889
I20260620 02:36:32.623273 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:36:32.780817 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 8770
Jun 20 02:36:32 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:36:32.799321 9177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:32.782062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.0.97.1:36527)
0620 02:36:32.782891 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:32.782898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:32.782932 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:36:32.782994 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:32.783001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:32.783220 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:32.783350 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:32.784693 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.785764 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.786998 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.787622 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.787725 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:32.792629 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:32.792645 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:32.792663 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:32.792703 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:32.795783 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:32.796733 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:32.796741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:32.796744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:32.796878 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:32.797268 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:32.797271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:32.797273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:32.797841 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:32.798229 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:32.798476 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 02:36:32.798731 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":400,"threads_started":1}
I20260620 02:36:32.800868 8893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:32.889734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 8770
I20260620 02:36:32.907040 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 8904
Jun 20 02:36:32 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:36:32.925925 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:32.908080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.0.97.2:37869)
0620 02:36:32.908665 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:32.908678 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:32.908769 (+ 91us) server_negotiation.cc:407] Connection header received
0620 02:36:32.908857 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:32.908864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:32.909108 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:32.909286 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:32.910814 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.912122 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.913491 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:32.914134 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:32.914260 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:32.919106 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:32.919122 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:32.919141 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:32.919183 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:32.922924 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:32.923791 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:32.923799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:32.923801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:32.924085 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:32.924454 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:32.924457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:32.924459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:32.924862 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:32.925089 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:32.925427 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 02:36:32.925631 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":257,"threads_started":1}
I20260620 02:36:32.927316 9027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:33.013986 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 8904
I20260620 02:36:33.028244 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 9038
Jun 20 02:36:33 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:36:33.046845 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:33.029408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.0.97.3:37779)
0620 02:36:33.030017 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:33.030024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:33.030055 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:36:33.030114 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:33.030120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:33.030362 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:33.030490 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:33.031740 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:33.032833 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:33.034122 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:33.034744 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:33.034848 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:33.039972 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:33.039990 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:33.040008 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:33.040051 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:33.043443 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:33.044467 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:33.044486 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:33.044489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:33.044643 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:33.045053 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:33.045056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:33.045058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:33.045466 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:33.045702 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:33.046097 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 02:36:33.046441 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":327,"threads_started":1}
I20260620 02:36:33.048462 9161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:33.134639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 9038
I20260620 02:36:33.151513 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 8677
I20260620 02:36:33.153005 8739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:33.257903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 8677
I20260620 02:36:33.280385 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45503
--webserver_interface=127.0.97.62
--webserver_port=40103
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:33.636341 9184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:33.636658 9184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:33.636751 9184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:33.644325 9184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:33.644454 9184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:33.644500 9184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:33.644550 9184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:33.644594 9184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:36:33.654652 9184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:33.656028 9184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:33.657354 9184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:33.670215 9190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:33.670146 9189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:33.670262 9192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:33.672009 9184 server_base.cc:1034] running on GCE node
I20260620 02:36:33.672868 9184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:33.674444 9184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:33.675731 9184 hybrid_clock.cc:648] HybridClock initialized: now 1781922993675673 us; error 58 us; skew 500 ppm
I20260620 02:36:33.676283 9184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:33 dist-test-slave-rpg6 krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:33.680115 9184 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:36:33.682314 9184 webserver.cc:460] Webserver started at http://127.0.97.62:40103/ using document root <none> and password file <none>
I20260620 02:36:33.683696 9184 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:33.684185 9184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:33.690753 9184 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 02:36:33.694381 9199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:33.696087 9184 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:36:33.696357 9184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "0f35cdb528ef426eaeec5c00e44b827a"
format_stamp: "Formatted at 2026-06-20 02:36:30 on dist-test-slave-rpg6"
server_key: "e482e9d8771e39416debdfdef3997649"
server_key_iv: "9fd131afae6be6af5fbb70eed2795e1d"
server_key_version: "encryptionkey@0"
I20260620 02:36:33.697431 9184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:33.722621 9184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:33.727654 9184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:33.728154 9184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:33.750677 9184 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45503
I20260620 02:36:33.750994 9250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45503 every 8 connection(s)
I20260620 02:36:33.752532 9184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:33.759395 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 9184
I20260620 02:36:33.760404 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36527
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45503
--webserver_port=40997
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:36:33.760782 9251 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:36:33.766696 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a: Bootstrap starting.
I20260620 02:36:33.788666 9251 log.cc:824] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:33.798722 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:36:33.799333 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a: Bootstrap complete.
I20260620 02:36:33.809079 9251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } }
I20260620 02:36:33.809962 9251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f35cdb528ef426eaeec5c00e44b827a, State: Initialized, Role: FOLLOWER
I20260620 02:36:33.810654 9251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } }
I20260620 02:36:33.810938 9251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:33.811064 9251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:33.811262 9251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:36:33.815322 9251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } }
I20260620 02:36:33.815982 9251 leader_election.cc:302] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f35cdb528ef426eaeec5c00e44b827a; no voters:
I20260620 02:36:33.816700 9251 leader_election.cc:288] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:36:33.816862 9255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:36:33.818228 9255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [term 2 LEADER]: Becoming Leader. State: Replica: 0f35cdb528ef426eaeec5c00e44b827a, State: Running, Role: LEADER
I20260620 02:36:33.819013 9255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } }
I20260620 02:36:33.819232 9251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:33.821894 9256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0f35cdb528ef426eaeec5c00e44b827a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } } }
I20260620 02:36:33.822831 9256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:33.822839 9257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f35cdb528ef426eaeec5c00e44b827a. Latest consensus state: current_term: 2 leader_uuid: "0f35cdb528ef426eaeec5c00e44b827a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f35cdb528ef426eaeec5c00e44b827a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45503 } } }
I20260620 02:36:33.824045 9257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:33.825342 9262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:33.831444 9262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:33.833141 9262 catalog_manager.cc:1259] Loaded cluster ID: 002346b9aee7401997201aee67d6ed7f
I20260620 02:36:33.833366 9262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:33.837829 9262 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:33.839650 9262 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f35cdb528ef426eaeec5c00e44b827a: Loaded TSK: 0
I20260620 02:36:33.841503 9262 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:36:34.143667 9253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:34.144090 9253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:34.144271 9253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:34.152220 9253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:34.152350 9253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:34.152524 9253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:36:34.163115 9253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40997
--tserver_master_addrs=127.0.97.62:45503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:34.164722 9253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:34.166224 9253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:34.185886 9277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:34.185845 9278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:34.185911 9280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:34.186571 9253 server_base.cc:1034] running on GCE node
I20260620 02:36:34.187417 9253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:34.188620 9253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:34.189896 9253 hybrid_clock.cc:648] HybridClock initialized: now 1781922994189834 us; error 60 us; skew 500 ppm
I20260620 02:36:34.190267 9253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922994, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:34.197996 9253 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:36:34.199812 9253 webserver.cc:460] Webserver started at http://127.0.97.1:40997/ using document root <none> and password file <none>
I20260620 02:36:34.200724 9253 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:34.200918 9253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:34.208664 9253 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:36:34.212024 9287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:34.214015 9253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:36:34.214244 9253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "d0f83e857c2c43fd92a3c657c94188c7"
format_stamp: "Formatted at 2026-06-20 02:36:31 on dist-test-slave-rpg6"
server_key: "5ce46ef7280bfc195a598acc8d3fecd7"
server_key_iv: "6e461bb28316a183729fd7bee1169ed1"
server_key_version: "encryptionkey@0"
I20260620 02:36:34.215340 9253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:34.241127 9253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:34.247769 9253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:34.248172 9253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:34.249433 9253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:34.251107 9253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:34.251263 9253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:34.251411 9253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:34.251507 9253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:34.297051 9253 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36527
I20260620 02:36:34.297112 9399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36527 every 8 connection(s)
I20260620 02:36:34.298614 9253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:36:34.299507 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 9253
I20260620 02:36:34.300423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37869
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45503
--webserver_port=42355
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:34.325757 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:34.302367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:37069)
0620 02:36:34.304674 (+ 2307us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:34.304727 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:34.307528 (+ 2801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:34.310016 (+ 2488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:34.310110 (+ 94us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:34.310989 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:34.311905 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.311938 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.313879 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.313887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.316283 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.316309 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.317588 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.317599 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.317941 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.318844 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:34.318891 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:34.324210 (+ 5319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1995,"thread_start_us":1799,"threads_started":1}
W20260620 02:36:34.325937 9402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:34.302114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37069 (local address 127.0.97.62:45503)
0620 02:36:34.302876 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:34.302888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:34.304888 (+ 2000us) server_negotiation.cc:407] Connection header received
0620 02:36:34.308457 (+ 3569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:34.308479 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:34.309081 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:34.309551 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:34.312265 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.313622 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.316502 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.317263 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.317436 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.324596 (+ 7160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37069: BlockingRecv error: recv got EOF from 127.0.97.1:37069 (error 108)
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":328,"threads_started":1}
W20260620 02:36:34.327142 9400 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45503 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45503: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:34.337805 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:34.328016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:51015)
0620 02:36:34.328247 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:34.328272 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:34.328420 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:34.328788 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:34.328795 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:34.329271 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:34.329468 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.329479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.330531 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.330537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.332090 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.332109 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.332869 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.332876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.333061 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.333750 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:34.333773 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:34.337471 (+ 3698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 02:36:34.337971 9402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:34.328109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51015 (local address 127.0.97.62:45503)
0620 02:36:34.328286 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:34.328303 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:34.328328 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:36:34.328563 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:34.328566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:34.328628 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:34.328790 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:34.329639 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.330379 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.332284 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.332711 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.332820 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.337706 (+ 4886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51015: BlockingRecv error: recv got EOF from 127.0.97.1:51015 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:34.347551 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:34.338615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:45969)
0620 02:36:34.338820 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:34.338846 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:34.338990 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:34.339327 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:34.339331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:34.339691 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:34.339930 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.339941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.341065 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.341071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.342023 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.342035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.342763 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.342769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.342952 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.343626 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:34.343643 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:34.347201 (+ 3558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 02:36:34.347676 9402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:34.338817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45969 (local address 127.0.97.62:45503)
0620 02:36:34.339012 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:34.339017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:34.339051 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:36:34.339123 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:34.339126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:34.339181 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:34.339302 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:34.340078 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.340919 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.342175 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.342590 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.342733 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.347435 (+ 4702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45969: BlockingRecv error: recv got EOF from 127.0.97.1:45969 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 02:36:34.347922 9400 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:36:34.655371 9404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:34.655742 9404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:34.655933 9404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:34.664337 9404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:34.664546 9404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:34.664741 9404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:36:34.675416 9404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42355
--tserver_master_addrs=127.0.97.62:45503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:34.676975 9404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:34.678373 9404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:34.693501 9409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:34.693581 9410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:34.693576 9412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:34.694229 9404 server_base.cc:1034] running on GCE node
I20260620 02:36:34.695027 9404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:34.696095 9404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:34.697319 9404 hybrid_clock.cc:648] HybridClock initialized: now 1781922994697270 us; error 40 us; skew 500 ppm
I20260620 02:36:34.697659 9404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922994, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:34.701407 9404 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:36:34.703068 9404 webserver.cc:460] Webserver started at http://127.0.97.2:42355/ using document root <none> and password file <none>
I20260620 02:36:34.703797 9404 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:34.704010 9404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:34.710021 9404 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:36:34.713217 9419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:34.714823 9404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 02:36:34.715029 9404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "8fac1228e48d46a68dfd8bcb3c903c2f"
format_stamp: "Formatted at 2026-06-20 02:36:31 on dist-test-slave-rpg6"
server_key: "763f0e62f09c39b9be85478ae301d061"
server_key_iv: "a32699d20f86ef4998aaff8981c03a06"
server_key_version: "encryptionkey@0"
I20260620 02:36:34.716171 9404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:34.736363 9404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:34.741406 9404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:34.741724 9404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:34.742996 9404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:34.744457 9404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:34.744551 9404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:34.744722 9404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:34.744807 9404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:34.780565 9404 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37869
I20260620 02:36:34.780607 9531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37869 every 8 connection(s)
I20260620 02:36:34.782074 9404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:36:34.791455 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 9404
I20260620 02:36:34.792187 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37779
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45503
--webserver_port=45819
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:36:34.800365 9402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:34.784895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43241 (local address 127.0.97.62:45503)
0620 02:36:34.785067 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:34.785071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:34.786335 (+ 1264us) server_negotiation.cc:407] Connection header received
0620 02:36:34.788495 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:34.788501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:34.788603 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:34.788728 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:34.790469 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.791459 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.793419 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.794301 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.794481 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.800055 (+ 5574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43241: BlockingRecv error: recv got EOF from 127.0.97.2:43241 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:36:34.801039 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:34.785285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:43241)
0620 02:36:34.786129 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:34.786178 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:34.788150 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:34.789001 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:34.789011 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:34.789587 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:34.790255 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.790276 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.791700 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.791715 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.793241 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.793256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.794615 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.794622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.794858 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.795671 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:34.795702 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:34.799670 (+ 3968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":279,"threads_started":1}
W20260620 02:36:34.801891 9532 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45503 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45503: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:34.813575 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:34.802570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:40213)
0620 02:36:34.802798 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:34.802821 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:34.802953 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:34.803386 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:34.803394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:34.803940 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:34.804174 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.804189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.805277 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.805286 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.807180 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.807200 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.808349 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.808360 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.808616 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.809333 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:34.809366 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:34.813241 (+ 3875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 02:36:34.813928 9402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:34.802718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40213 (local address 127.0.97.62:45503)
0620 02:36:34.802913 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:34.802918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:34.802935 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:36:34.803084 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:34.803087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:34.803146 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:34.803227 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:34.804359 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.805107 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.807413 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.808153 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.808277 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.813485 (+ 5208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40213: BlockingRecv error: recv got EOF from 127.0.97.2:40213 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:34 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:34.824828 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:34.814474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:54985)
0620 02:36:34.814677 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:34.814694 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:34.814791 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:34.815273 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:34.815278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:34.815618 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:34.815774 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.815783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.817708 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.817713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.818702 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:34.818716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.820098 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.820105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:34.820340 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.821144 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:34.821168 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:34.824476 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 02:36:34.824956 9402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:34.814513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54985 (local address 127.0.97.62:45503)
0620 02:36:34.814964 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:34.814968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:34.814983 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:36:34.815029 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:34.815032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:34.815086 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:34.815208 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:34.815995 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.817512 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.818914 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:34.819910 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:34.820088 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:34.824732 (+ 4644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54985: BlockingRecv error: recv got EOF from 127.0.97.2:54985 (error 108)
Metrics: {"server-negotiator.queue_time_us":335}
W20260620 02:36:34.825614 9532 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:36:35.155637 9535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:35.156096 9535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:35.156284 9535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:35.164114 9535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:35.164289 9535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:35.164480 9535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:36:35.175284 9535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45819
--tserver_master_addrs=127.0.97.62:45503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:35.176815 9535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:35.178236 9535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:35.192984 9541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:35.192930 9543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:35.193014 9540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:35.193975 9535 server_base.cc:1034] running on GCE node
I20260620 02:36:35.194766 9535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:35.196029 9535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:35.197392 9535 hybrid_clock.cc:648] HybridClock initialized: now 1781922995197288 us; error 56 us; skew 500 ppm
I20260620 02:36:35.198027 9535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:35 dist-test-slave-rpg6 krb5kdc[8664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781922995, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:35.201974 9535 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:36:35.203799 9535 webserver.cc:460] Webserver started at http://127.0.97.3:45819/ using document root <none> and password file <none>
I20260620 02:36:35.204649 9535 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:35.204828 9535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:35.210901 9535 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 02:36:35.214186 9550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:35.215974 9535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:36:35.216188 9535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "f5b667fa96de4f6093058fb04159d446"
format_stamp: "Formatted at 2026-06-20 02:36:32 on dist-test-slave-rpg6"
server_key: "31b1c2b7f03447f9e35d23e785a639f0"
server_key_iv: "7e6f2a2d13b7ac5b33baca22d80a8f2a"
server_key_version: "encryptionkey@0"
I20260620 02:36:35.217360 9535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:35.280393 9535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:35.286020 9535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:35.286355 9535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:35.287495 9535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:35.289042 9535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:35.289188 9535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:35.289320 9535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:35.289409 9535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:35.330767 9535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37779
I20260620 02:36:35.330863 9662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37779 every 8 connection(s)
I20260620 02:36:35.332453 9535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:36:35.334113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 9535
Jun 20 02:36:35 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:35 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:36:35.354671 9666 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:35.335960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39033 (local address 127.0.97.62:45503)
0620 02:36:35.338652 (+ 2692us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:35.338659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:35.338688 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:36:35.340985 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:35.340989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:35.341095 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:35.341244 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:35.343173 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.344411 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.347200 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.347735 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.347947 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:35.354308 (+ 6361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39033: BlockingRecv error: recv got EOF from 127.0.97.3:39033 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":2497,"thread_start_us":2381,"threads_started":1}
I20260620 02:36:35.355581 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:35.336339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:39033)
0620 02:36:35.337628 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:35.337702 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:35.340538 (+ 2836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:35.341487 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:35.341497 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:35.342068 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:35.342950 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:35.342979 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.344644 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.344650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:35.346978 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:35.347002 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.348040 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.348055 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:35.348347 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:35.349175 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:35.349242 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:35.354073 (+ 4831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":457,"threads_started":1}
W20260620 02:36:35.356334 9663 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45503 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45503: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260620 02:36:35.356734 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:35.335971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.0.97.62:45503)
0620 02:36:35.338771 (+ 2800us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:35.338775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:35.338787 (+ 12us) server_negotiation.cc:407] Connection header received
0620 02:36:35.338837 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:35.338841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:35.338903 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:35.339010 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:35.341138 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.342112 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.343741 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.344550 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.344658 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:35.348299 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:35.348308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:35.348328 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:35.348364 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:35.353050 (+ 4686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:35.353797 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:35.353805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:35.353812 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:35.354001 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:35.354418 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:35.354422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:35.354424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:35.355006 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:35.355424 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:35.355899 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 02:36:35.356275 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":2683,"thread_start_us":2575,"threads_started":1}
Jun 20 02:36:35 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:35 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:35 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:35.365007 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:35.348589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:35647)
0620 02:36:35.350524 (+ 1935us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:35.350545 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:35.350681 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:35.351400 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:35.351406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:35.352003 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:35.352253 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:35.352265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.357137 (+ 4872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.357147 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:35.358861 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:35.358878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.359801 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.359810 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:35.360019 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:35.360724 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:35.360742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:35.364635 (+ 3893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":284,"threads_started":1}
W20260620 02:36:35.365267 9670 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:35.348679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35647 (local address 127.0.97.62:45503)
0620 02:36:35.351019 (+ 2340us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:35.351024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:35.351040 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:36:35.351088 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:35.351092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:35.351201 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:35.351334 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:35.352447 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.356938 (+ 4491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.359026 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.359595 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.359729 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:35.364932 (+ 5203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35647: BlockingRecv error: recv got EOF from 127.0.97.1:35647 (error 108)
Metrics: {"server-negotiator.queue_time_us":2235,"thread_start_us":1689,"threads_started":1}
Jun 20 02:36:35 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:35.366693 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:35.357037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:52905)
0620 02:36:35.357353 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:35.357375 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:35.357484 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:35.357787 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:35.357793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:35.358148 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:35.358286 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:35.358294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.359507 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.359515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:35.361146 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:35.361164 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.362203 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.362210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:35.362432 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:35.363108 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:35.363128 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:35.366488 (+ 3360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171}
W20260620 02:36:35.367024 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:35.357156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52905 (local address 127.0.97.62:45503)
0620 02:36:35.357408 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:35.357412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:35.357427 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:36:35.357597 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:35.357600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:35.357649 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:35.357757 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:35.358422 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.359354 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.361324 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.362025 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.362144 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:35.366724 (+ 4580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52905: BlockingRecv error: recv got EOF from 127.0.97.3:52905 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Jun 20 02:36:35 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:35 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:35.375304 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:35.367398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:48625)
0620 02:36:35.367618 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:35.367640 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:35.367768 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:35.368201 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:35.368204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:35.368484 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:35.368640 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:35.368649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.369670 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.369673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:35.370569 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:35.370580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.371317 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.371322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:35.371457 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:35.372105 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:35.372122 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:35.374949 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 02:36:35.375401 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:35.367473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48625 (local address 127.0.97.62:45503)
0620 02:36:35.367632 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:35.367640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:35.367696 (+ 56us) server_negotiation.cc:407] Connection header received
0620 02:36:35.368004 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:35.368008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:35.368067 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:35.368147 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:35.368782 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.369541 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.370698 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.371155 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.371261 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:35.375141 (+ 3880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48625: BlockingRecv error: recv got EOF from 127.0.97.3:48625 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 02:36:35.375660 9663 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 02:36:35 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:35 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:35.837899 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:35.826340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:54715)
0620 02:36:35.826953 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:35.826970 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:35.827088 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:35.827427 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:35.827430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:35.827981 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:35.828209 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:35.828220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.829316 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.829545 (+ 229us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:35.831042 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:35.831061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.832025 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.832031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:35.832194 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:35.833043 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:35.833073 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:35.837554 (+ 4481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":333,"threads_started":1}
W20260620 02:36:35.838052 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:35.826320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54715 (local address 127.0.97.62:45503)
0620 02:36:35.826563 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:35.826568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:35.827071 (+ 503us) server_negotiation.cc:407] Connection header received
0620 02:36:35.827194 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:35.827198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:35.827260 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:35.827351 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:35.828367 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.829145 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.831261 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:35.831702 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:35.831814 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:35.837820 (+ 6006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54715: BlockingRecv error: recv got EOF from 127.0.97.2:54715 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 02:36:36 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:36 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:36.376583 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:36.365970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:34477)
0620 02:36:36.366615 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:36.366631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:36.366799 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:36.367390 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:36.367405 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:36.367916 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:36.368088 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:36.368098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.369316 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.369321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:36.370262 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:36.370276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.371045 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.371050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:36.371161 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:36.372198 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:36.372225 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:36.376223 (+ 3998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":377,"threads_started":1}
W20260620 02:36:36.376741 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:36.366330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34477 (local address 127.0.97.62:45503)
0620 02:36:36.366973 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:36.366979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:36.366999 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:36:36.367068 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:36.367071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:36.367150 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:36.367295 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:36.368344 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.369070 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.370450 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.370885 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.370988 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:36.376449 (+ 5461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34477: BlockingRecv error: recv got EOF from 127.0.97.1:34477 (error 108)
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":397,"threads_started":1}
Jun 20 02:36:36 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:36 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:36.385753 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:36.376339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:42699)
0620 02:36:36.377435 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:36.377457 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:36.377595 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:36.377950 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:36.377955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:36.378416 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:36.378595 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:36.378603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.379668 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.379673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:36.380718 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:36.380730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.381467 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.381476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:36.381677 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:36.382295 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:36.382314 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:36.385508 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":829,"threads_started":1}
W20260620 02:36:36.386054 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:36.376844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42699 (local address 127.0.97.62:45503)
0620 02:36:36.377305 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:36.377312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:36.377543 (+ 231us) server_negotiation.cc:407] Connection header received
0620 02:36:36.377745 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:36.377757 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:36.377819 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:36.377928 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:36.378747 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.379485 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.380873 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.381275 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.381428 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:36.385748 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42699: BlockingRecv error: recv got EOF from 127.0.97.3:42699 (error 108)
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":581,"threads_started":1}
Jun 20 02:36:36 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:36 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:36.851596 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:36.838902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:51823)
0620 02:36:36.839467 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:36.839484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:36.839613 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:36.839969 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:36.839972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:36.840362 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:36.840509 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:36.840517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.841604 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.841608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:36.842542 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:36.842554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.843496 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.843501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:36.843625 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:36.848432 (+ 4807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:36.848454 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:36.851261 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":318,"threads_started":1}
W20260620 02:36:36.852032 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:36.839056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51823 (local address 127.0.97.62:45503)
0620 02:36:36.839244 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:36.839250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:36.839610 (+ 360us) server_negotiation.cc:407] Connection header received
0620 02:36:36.839727 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:36.839731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:36.839808 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:36.839922 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:36.840666 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.841438 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.842888 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:36.843319 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:36.843430 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:36.851706 (+ 8276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51823: BlockingRecv error: recv got EOF from 127.0.97.2:51823 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 02:36:37 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:37 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:37.387882 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:37.377702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:59127)
0620 02:36:37.378855 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:37.378878 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:37.379019 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:37.379482 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:37.379486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:37.379866 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:37.380048 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:37.380057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.381115 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.381120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:37.382083 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:37.382096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.382963 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.382967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:37.383110 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:37.384072 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:37.384126 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:37.387423 (+ 3297us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1032,"thread_start_us":290,"threads_started":1}
W20260620 02:36:37.388053 9678 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:37.378249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59127 (local address 127.0.97.62:45503)
0620 02:36:37.378779 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:37.378785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:37.379090 (+ 305us) server_negotiation.cc:407] Connection header received
0620 02:36:37.379159 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:37.379165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:37.379322 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:37.379420 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:37.380213 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.380961 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.382264 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.382796 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.382912 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:37.387691 (+ 4779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59127: BlockingRecv error: recv got EOF from 127.0.97.1:59127 (error 108)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":238,"threads_started":1}
Jun 20 02:36:37 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:37 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:37.396672 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:37.386902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:50729)
0620 02:36:37.387785 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:37.387806 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:37.387985 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:37.388517 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:37.388521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:37.388875 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:37.389014 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:37.389022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.390350 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.390360 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:37.391751 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:37.391767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.392718 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.392724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:37.392841 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:37.393458 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:37.393475 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:37.396403 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":628,"threads_started":1}
W20260620 02:36:37.396953 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:37.386995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50729 (local address 127.0.97.62:45503)
0620 02:36:37.388143 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:37.388150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:37.388170 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:36:37.388230 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:37.388234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:37.388325 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:37.388458 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:37.389201 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.390049 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.391951 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.392533 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.392662 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:37.396671 (+ 4009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50729: BlockingRecv error: recv got EOF from 127.0.97.3:50729 (error 108)
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":310,"threads_started":1}
Jun 20 02:36:37 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:37 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:37.863754 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:37.852848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:53931)
0620 02:36:37.853366 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:37.853385 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:37.853528 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:37.853845 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:37.853848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:37.854174 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:37.854390 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:37.854403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.856220 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.856230 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:37.857332 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:37.857358 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.858454 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.858460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:37.858678 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:37.859635 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:37.859659 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:37.863417 (+ 3758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":409,"threads_started":1}
W20260620 02:36:37.864130 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:37.852949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53931 (local address 127.0.97.62:45503)
0620 02:36:37.853256 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:37.853262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:37.853497 (+ 235us) server_negotiation.cc:407] Connection header received
0620 02:36:37.853645 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:37.853648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:37.853711 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:37.853800 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:37.854799 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.855951 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.857622 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:37.858172 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:37.858307 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:37.863771 (+ 5464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53931: BlockingRecv error: recv got EOF from 127.0.97.2:53931 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 02:36:38 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:38 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:38.399747 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:38.389246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:53609)
0620 02:36:38.389786 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:38.389804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:38.389931 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:38.390574 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:38.390578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:38.390961 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:38.391133 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:38.391142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.392599 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.392604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:38.393658 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:38.393673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.394563 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.394568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:38.394720 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:38.395451 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:38.395470 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:38.399394 (+ 3924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":306,"threads_started":1}
W20260620 02:36:38.399959 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:38.389341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53609 (local address 127.0.97.62:45503)
0620 02:36:38.389861 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:38.389865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:38.389884 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:36:38.390212 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:38.390218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:38.390354 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:38.390509 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:38.391352 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.392433 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.393856 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.394400 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.394516 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:38.399638 (+ 5122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53609: BlockingRecv error: recv got EOF from 127.0.97.1:53609 (error 108)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":282,"threads_started":1}
Jun 20 02:36:38 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:38 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:38.408257 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:38.397675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:34615)
0620 02:36:38.398210 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:38.398230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:38.398352 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:38.399291 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:38.399296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:38.399702 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:38.399968 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:38.399979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.401235 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.401248 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:38.402275 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:38.402289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.403049 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.403055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:38.403257 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:38.404181 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:38.404200 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:38.407841 (+ 3641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":315,"threads_started":1}
W20260620 02:36:38.408600 9685 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:38.398293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34615 (local address 127.0.97.62:45503)
0620 02:36:38.398789 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:38.398794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:38.398813 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:36:38.398863 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:38.398868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:38.398940 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:38.399056 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:38.400158 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.401064 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.402436 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.402890 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.403034 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:38.408133 (+ 5099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34615: BlockingRecv error: recv got EOF from 127.0.97.3:34615 (error 108)
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":246,"threads_started":1}
Jun 20 02:36:38 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:38 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:38.874580 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:38.865134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:37761)
0620 02:36:38.865776 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:38.865799 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:38.865930 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:38.866381 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:38.866388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:38.866873 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:38.867095 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:38.867107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.868574 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.868579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:38.869556 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:38.869569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.870251 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.870256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:38.870414 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:38.871084 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:38.871106 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:38.874334 (+ 3228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":406,"threads_started":1}
W20260620 02:36:38.874876 9685 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:38.865210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37761 (local address 127.0.97.62:45503)
0620 02:36:38.865453 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:38.865458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:38.865941 (+ 483us) server_negotiation.cc:407] Connection header received
0620 02:36:38.866039 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:38.866044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:38.866124 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:38.866329 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:38.867357 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.868437 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.869712 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:38.870093 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:38.870191 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:38.874585 (+ 4394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37761: BlockingRecv error: recv got EOF from 127.0.97.2:37761 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Jun 20 02:36:39 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:39 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:39 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:39 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:36:39.420001 9688 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:39.401196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38407 (local address 127.0.97.62:45503)
0620 02:36:39.402996 (+ 1800us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:39.403000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:39.403016 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:36:39.403244 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:39.403248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:39.403310 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:39.403407 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:39.404656 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.405767 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.408337 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.409026 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.409204 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:39.419684 (+ 10480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38407: BlockingRecv error: recv got EOF from 127.0.97.1:38407 (error 108)
Metrics: {"server-negotiator.queue_time_us":1697,"thread_start_us":1609,"threads_started":1}
I20260620 02:36:39.420034 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:39.401141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:38407)
0620 02:36:39.402466 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:39.402493 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:39.403066 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:39.403478 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:39.403484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:39.404106 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:39.404370 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:39.404386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.406071 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.406080 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:39.408088 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:39.408115 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.409232 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.409238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:39.409497 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:39.415245 (+ 5748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:39.415279 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:39.419474 (+ 4195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1163,"thread_start_us":948,"threads_started":1}
I20260620 02:36:39.420624 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:39.409363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:38435)
0620 02:36:39.410878 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:39.410893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:39.411055 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:39.411401 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:39.411406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:39.411800 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:39.412013 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:39.412022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.413147 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.413154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:39.414209 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:39.414225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.415313 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.415319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:39.415733 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:39.416385 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:39.416402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:39.420214 (+ 3812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1415,"thread_start_us":1310,"threads_started":1}
W20260620 02:36:39.421232 9690 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:39.409586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38435 (local address 127.0.97.62:45503)
0620 02:36:39.410757 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:39.410763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:39.410986 (+ 223us) server_negotiation.cc:407] Connection header received
0620 02:36:39.411192 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:39.411202 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:39.411261 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:39.411387 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:39.412160 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.412984 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.414362 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.414897 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.415030 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:39.420845 (+ 5815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38435: BlockingRecv error: recv got EOF from 127.0.97.3:38435 (error 108)
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":862,"threads_started":1}
Jun 20 02:36:39 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:39 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:39.886967 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:39.875905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:60945)
0620 02:36:39.877013 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:39.877041 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:39.877248 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:39.877811 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:39.877818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:39.878380 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:39.878610 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:39.878622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.880157 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.880164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:39.881197 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:39.881211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.881939 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.881945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:39.882127 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:39.882774 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:39.882797 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:39.886593 (+ 3796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":656,"threads_started":1}
W20260620 02:36:39.887104 9690 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:39.876558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60945 (local address 127.0.97.62:45503)
0620 02:36:39.876862 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:39.876868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:39.877181 (+ 313us) server_negotiation.cc:407] Connection header received
0620 02:36:39.877511 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:39.877515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:39.877585 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:39.877677 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:39.878783 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.880029 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.881357 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:39.881783 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:39.881925 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:39.886853 (+ 4928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60945: BlockingRecv error: recv got EOF from 127.0.97.2:60945 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Jun 20 02:36:40 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:40 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:40 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:40 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:40.432327 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:40.421022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:45153)
0620 02:36:40.422446 (+ 1424us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:40.422461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:40.422640 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:40.422975 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:40.422982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:40.423391 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:40.423556 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:40.423565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.424794 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.424800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:40.425759 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:40.425771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.426745 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.426752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:40.426960 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:40.427964 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:40.427983 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:40.431959 (+ 3976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":668,"threads_started":1}
I20260620 02:36:40.432541 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:40.421694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:40143)
0620 02:36:40.422249 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:40.422269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:40.422409 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:40.422842 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:40.422847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:40.423400 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:40.423561 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:40.423570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.424764 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.424770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:40.425738 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:40.425750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.426691 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.426697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:40.426914 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:40.427799 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:40.427819 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:40.432211 (+ 4392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":311,"threads_started":1}
W20260620 02:36:40.432529 9693 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:40.421109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45153 (local address 127.0.97.62:45503)
0620 02:36:40.421718 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:40.421727 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:40.422565 (+ 838us) server_negotiation.cc:407] Connection header received
0620 02:36:40.422776 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:40.422780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:40.422836 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:40.422945 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:40.423713 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.424652 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.425922 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.426558 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.426731 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:40.432220 (+ 5489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45153: BlockingRecv error: recv got EOF from 127.0.97.1:45153 (error 108)
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":455,"thread_start_us":285,"threads_started":1}
W20260620 02:36:40.432814 9695 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:40.421918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40143 (local address 127.0.97.62:45503)
0620 02:36:40.422550 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:40.422556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:40.422577 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:36:40.422624 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:40.422627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:40.422691 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:40.422813 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:40.423713 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.424605 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.425929 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.426525 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.426663 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:40.432478 (+ 5815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40143: BlockingRecv error: recv got EOF from 127.0.97.3:40143 (error 108)
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":483,"thread_start_us":359,"threads_started":1}
Jun 20 02:36:40 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:40 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:40.897228 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:40.888158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:43351)
0620 02:36:40.888826 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:40.888843 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:40.888972 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:40.889457 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:40.889461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:40.889879 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:40.890040 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:40.890050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.891116 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.891122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:40.892158 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:40.892171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.892994 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.892999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:40.893215 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:40.893899 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:40.893918 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:40.896935 (+ 3017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":412,"threads_started":1}
W20260620 02:36:40.897555 9695 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:40.888298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43351 (local address 127.0.97.62:45503)
0620 02:36:40.888481 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:40.888486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:40.888934 (+ 448us) server_negotiation.cc:407] Connection header received
0620 02:36:40.889207 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:40.889211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:40.889284 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:40.889394 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:40.890239 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.890967 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.892377 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:40.892809 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:40.892923 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:40.897293 (+ 4370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43351: BlockingRecv error: recv got EOF from 127.0.97.2:43351 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 02:36:41 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:41 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:41 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:41 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:41.445745 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:41.433730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:52517)
0620 02:36:41.434509 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:41.434536 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:41.434683 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:41.435586 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:41.435591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:41.436072 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:41.436314 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:41.436328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.437861 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.437865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:41.439354 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:41.439370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.440500 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.440506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:41.440672 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:41.441521 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:41.441546 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:41.445385 (+ 3839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":401,"threads_started":1}
I20260620 02:36:41.445816 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:41.433893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:35711)
0620 02:36:41.434805 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:41.434831 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:41.434918 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:41.436252 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:41.436258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:41.436649 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:41.436791 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:41.436799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.438260 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.438265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:41.439679 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:41.439691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.440853 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.440861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:41.441058 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:41.441904 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:41.441921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:41.445430 (+ 3509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":713,"threads_started":1}
W20260620 02:36:41.446175 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:41.435198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35711 (local address 127.0.97.62:45503)
0620 02:36:41.435958 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:41.435963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:41.435978 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:36:41.436015 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:41.436018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:41.436073 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:41.436164 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:41.436949 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.438082 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.439860 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.440677 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.440794 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:41.445803 (+ 5009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35711: BlockingRecv error: recv got EOF from 127.0.97.3:35711 (error 108)
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":599,"thread_start_us":270,"threads_started":1}
W20260620 02:36:41.446539 9698 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:41.433856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52517 (local address 127.0.97.62:45503)
0620 02:36:41.435181 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:41.435187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:41.435207 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:36:41.435278 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:41.435283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:41.435361 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:41.435496 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:41.436506 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.437688 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.439594 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.440297 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.440445 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:41.445774 (+ 5329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52517: BlockingRecv error: recv got EOF from 127.0.97.1:52517 (error 108)
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":1181,"thread_start_us":1166,"threads_started":1}
Jun 20 02:36:41 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:41 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:41.907800 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:41.898321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:56147)
0620 02:36:41.899013 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:41.899030 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:41.899172 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:41.899809 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:41.899815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:41.900345 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:41.900527 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:41.900536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.901795 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.901799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:41.902868 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:41.902882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.903727 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.903733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:41.903936 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:41.904629 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:41.904648 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:41.907545 (+ 2897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":284,"threads_started":1}
W20260620 02:36:41.908231 9698 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:41.898898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56147 (local address 127.0.97.62:45503)
0620 02:36:41.899102 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:41.899107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:41.899126 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:36:41.899466 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:41.899472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:41.899562 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:41.899666 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:41.900684 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.901587 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.903110 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:41.903549 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:41.903670 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:41.907884 (+ 4214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56147: BlockingRecv error: recv got EOF from 127.0.97.2:56147 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 02:36:42 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:42 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:42 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:42 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:42.457033 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:42.446951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:49003)
0620 02:36:42.447530 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:42.447555 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:42.447697 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:42.448428 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:42.448432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:42.448798 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:42.448942 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:42.448950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.450017 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.450025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:42.450960 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:42.450972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.451685 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.451691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:42.451923 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:42.452644 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:42.452894 (+ 250us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:42.456748 (+ 3854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":291,"threads_started":1}
I20260620 02:36:42.457437 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:42.446959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:58659)
0620 02:36:42.448174 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:42.448193 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:42.448308 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:42.448785 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:42.448791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:42.449445 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:42.449608 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:42.449617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.450777 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.450784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:42.451695 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:42.451706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.452592 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.452598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:42.452786 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:42.453372 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:42.453395 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:42.457107 (+ 3712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1080,"thread_start_us":886,"threads_started":1}
W20260620 02:36:42.457458 9704 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:42.447042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49003 (local address 127.0.97.62:45503)
0620 02:36:42.447791 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:42.447799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:42.447820 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:36:42.447907 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:42.447915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:42.447995 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:42.448098 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:42.449097 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.449876 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.451105 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.451497 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.451637 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:42.457042 (+ 5405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49003: BlockingRecv error: recv got EOF from 127.0.97.1:49003 (error 108)
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":510,"threads_started":1}
W20260620 02:36:42.458072 9705 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:42.447747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58659 (local address 127.0.97.62:45503)
0620 02:36:42.448509 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:42.448513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:42.448528 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:36:42.448566 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:42.448569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:42.448622 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:42.448750 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:42.449781 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.450628 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.451875 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.452439 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.452557 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:42.457725 (+ 5168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58659: BlockingRecv error: recv got EOF from 127.0.97.3:58659 (error 108)
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":539,"threads_started":1}
Jun 20 02:36:42 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:42 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:42.917954 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:42.908889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:43471)
0620 02:36:42.909413 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:42.909433 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:42.909557 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:42.909859 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:42.909862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:42.910217 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:42.910359 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:42.910367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.911577 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.911581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:42.912627 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:42.912638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.913418 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.913423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:42.913533 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:42.914368 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:42.914387 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:42.917687 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":319,"threads_started":1}
W20260620 02:36:42.918233 9705 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:42.909053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43471 (local address 127.0.97.62:45503)
0620 02:36:42.909252 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:42.909257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:42.909519 (+ 262us) server_negotiation.cc:407] Connection header received
0620 02:36:42.909662 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:42.909665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:42.909727 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:42.909816 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:42.910491 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.911432 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.912790 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:42.913227 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:42.913346 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:42.917964 (+ 4618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43471: BlockingRecv error: recv got EOF from 127.0.97.2:43471 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 02:36:43 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:43 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:43 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:43 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:43.469681 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:43.458798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:56897)
0620 02:36:43.459439 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:43.459468 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:43.459700 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:43.461250 (+ 1550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:43.461255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:43.461764 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:43.461933 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:43.461948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.463046 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.463051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:43.464008 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:43.464019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.464848 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.464855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:43.465031 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:43.465710 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:43.465736 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:43.469340 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":321,"threads_started":1}
W20260620 02:36:43.469864 9710 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:43.460292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56897 (local address 127.0.97.62:45503)
0620 02:36:43.460796 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:43.460802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:43.460819 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:36:43.460874 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:43.460881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:43.460969 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:43.461090 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:43.462099 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.462860 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.464166 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.464603 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.464807 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:43.469579 (+ 4772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56897: BlockingRecv error: recv got EOF from 127.0.97.3:56897 (error 108)
Metrics: {"server-negotiator.queue_time_us":380,"spinlock_wait_cycles":10112,"thread_start_us":250,"threads_started":1}
I20260620 02:36:43.470361 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:43.458798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:38529)
0620 02:36:43.460821 (+ 2023us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:43.460837 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:43.460999 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:43.461411 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:43.461416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:43.462006 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:43.462172 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:43.462181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.463283 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.463290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:43.464303 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:43.464315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.465149 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.465156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:43.465340 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:43.466145 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:43.466172 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:43.470042 (+ 3870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1558,"thread_start_us":1435,"threads_started":1}
W20260620 02:36:43.470726 9709 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:43.459047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38529 (local address 127.0.97.62:45503)
0620 02:36:43.460651 (+ 1604us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:43.460656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:43.460940 (+ 284us) server_negotiation.cc:407] Connection header received
0620 02:36:43.461129 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:43.461136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:43.461249 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:43.461348 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:43.462343 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.463135 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.464451 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.464979 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.465119 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:43.470342 (+ 5223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38529: BlockingRecv error: recv got EOF from 127.0.97.1:38529 (error 108)
Metrics: {"server-negotiator.queue_time_us":1461,"spinlock_wait_cycles":768,"thread_start_us":1074,"threads_started":1}
Jun 20 02:36:43 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:43 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:43.929304 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:43.919327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:47409)
0620 02:36:43.920004 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:43.920038 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:43.920207 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:43.920681 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:43.920685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:43.921349 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:43.921556 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:43.921564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.922916 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.922922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:43.924046 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:43.924060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.924778 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.924784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:43.924976 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:43.925642 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:43.925671 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:43.929031 (+ 3360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":300,"threads_started":1}
W20260620 02:36:43.929780 9709 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:43.919452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47409 (local address 127.0.97.62:45503)
0620 02:36:43.919752 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:43.919760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:43.920136 (+ 376us) server_negotiation.cc:407] Connection header received
0620 02:36:43.920415 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:43.920422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:43.920513 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:43.920632 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:43.921699 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.922772 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.924201 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:43.924612 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:43.924765 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:43.929531 (+ 4766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47409: BlockingRecv error: recv got EOF from 127.0.97.2:47409 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Jun 20 02:36:44 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:44 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:44 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:44 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:44.484498 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:44.471396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:53191)
0620 02:36:44.472289 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:44.472315 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:44.472409 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:44.472842 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:44.472847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:44.473306 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:44.473439 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:44.473448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.474765 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.474770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:44.476156 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:44.476172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.477016 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.477024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:44.477198 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:44.480951 (+ 3753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:44.480974 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:44.484157 (+ 3183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":625,"threads_started":1}
W20260620 02:36:44.484732 9715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:44.471935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53191 (local address 127.0.97.62:45503)
0620 02:36:44.472571 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:44.472575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:44.472589 (+ 14us) server_negotiation.cc:407] Connection header received
0620 02:36:44.472626 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:44.472629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:44.472675 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:44.472760 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:44.473585 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.474624 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.476318 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.476819 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.476976 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:44.484462 (+ 7486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53191: BlockingRecv error: recv got EOF from 127.0.97.1:53191 (error 108)
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":422,"threads_started":1}
I20260620 02:36:44.484787 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:44.470859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:43347)
0620 02:36:44.471524 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:44.471546 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:44.471767 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:44.472363 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:44.472369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:44.472893 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:44.473120 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:44.473131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.474424 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.474432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:44.475432 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:44.475447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.476435 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.476441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:44.476638 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:44.480986 (+ 4348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:44.481008 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:44.484461 (+ 3453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":377,"threads_started":1}
W20260620 02:36:44.485074 9713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:44.471123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43347 (local address 127.0.97.62:45503)
0620 02:36:44.471939 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:44.471946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:44.471967 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:36:44.472039 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:44.472046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:44.472153 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:44.472879 (+ 726us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:44.473320 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.474265 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.475585 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.476285 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.476402 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:44.484769 (+ 8367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43347: BlockingRecv error: recv got EOF from 127.0.97.3:43347 (error 108)
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":548,"threads_started":1}
Jun 20 02:36:44 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:44 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:44.939102 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:44.930292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:41203)
0620 02:36:44.930800 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:44.930822 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:44.930941 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:44.931332 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:44.931335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:44.931731 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:44.931971 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:44.931982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.933056 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.933061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:44.934060 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:44.934073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.934972 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.934977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:44.935269 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:44.936057 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:44.936075 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:44.938879 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":310,"threads_started":1}
W20260620 02:36:44.939616 9713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:44.930461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41203 (local address 127.0.97.62:45503)
0620 02:36:44.930644 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:44.930651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:44.930910 (+ 259us) server_negotiation.cc:407] Connection header received
0620 02:36:44.931049 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:44.931052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:44.931111 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:44.931256 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:44.932132 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.932905 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.934326 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:44.934812 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:44.934922 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:44.939363 (+ 4441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41203: BlockingRecv error: recv got EOF from 127.0.97.2:41203 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 02:36:45 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:45 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:45 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:45 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:45.498791 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:45.486050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:33383)
0620 02:36:45.487280 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:45.487302 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:45.487419 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:45.488110 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:45.488116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:45.488567 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:45.488722 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:45.488732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.490210 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.490215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:45.491723 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:45.491742 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.493197 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.493204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:45.493372 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:45.494191 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:45.494218 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:45.498407 (+ 4189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1121,"thread_start_us":1000,"threads_started":1}
I20260620 02:36:45.499275 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:45.485836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:54987)
0620 02:36:45.486471 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:45.486500 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:45.486691 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:45.487484 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:45.487489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:45.488046 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:45.488321 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:45.488336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.489923 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.489929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:45.491738 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:45.491762 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.493721 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.493728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:45.493957 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:45.494785 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:45.494810 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:45.499022 (+ 4212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":331,"threads_started":1}
W20260620 02:36:45.499440 9720 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:45.486657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33383 (local address 127.0.97.62:45503)
0620 02:36:45.487847 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:45.487852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:45.487865 (+ 13us) server_negotiation.cc:407] Connection header received
0620 02:36:45.487900 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:45.487903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:45.487958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:45.488040 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:45.488900 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.490000 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.492000 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.492742 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.492898 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:45.498738 (+ 5840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33383: BlockingRecv error: recv got EOF from 127.0.97.3:33383 (error 108)
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":979,"threads_started":1}
W20260620 02:36:45.499804 9719 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:45.486130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54987 (local address 127.0.97.62:45503)
0620 02:36:45.486700 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:45.486706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:45.486731 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:36:45.486848 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:45.486856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:45.486968 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:45.487250 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:45.488509 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.489704 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.491985 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.493157 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.493308 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:45.499423 (+ 6115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54987: BlockingRecv error: recv got EOF from 127.0.97.1:54987 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":459,"thread_start_us":338,"threads_started":1}
Jun 20 02:36:45 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:45 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:45.950019 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:45.940128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:47811)
0620 02:36:45.940780 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:45.940798 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:45.940924 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:45.941366 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:45.941369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:45.941800 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:45.941969 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:45.941977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.943106 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.943111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:45.944223 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:45.944243 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.945039 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.945045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:45.945196 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:45.945909 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:45.945928 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:45.949655 (+ 3727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":383,"threads_started":1}
W20260620 02:36:45.950111 9719 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:45.940429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47811 (local address 127.0.97.62:45503)
0620 02:36:45.940672 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:45.940678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:45.940913 (+ 235us) server_negotiation.cc:407] Connection header received
0620 02:36:45.941068 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:45.941074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:45.941151 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:45.941279 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:45.942145 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.942917 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.944417 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:45.944839 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:45.944960 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:45.949878 (+ 4918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47811: BlockingRecv error: recv got EOF from 127.0.97.2:47811 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 20 02:36:46 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:46 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:46 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:46 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:46.511546 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:46.499888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:40223)
0620 02:36:46.501342 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:46.501362 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:46.501502 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:46.501844 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:46.501849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:46.502417 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:46.502672 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:46.502683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.503880 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.503887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:46.505073 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:46.505087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.506328 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.506336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:46.506527 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:46.507411 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:46.507429 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:46.511157 (+ 3728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1317,"thread_start_us":1182,"threads_started":1}
W20260620 02:36:46.511777 9723 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:46.499968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40223 (local address 127.0.97.62:45503)
0620 02:36:46.500479 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:46.500484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:46.501486 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 02:36:46.501621 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:46.501625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:46.501687 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:46.501779 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:46.502823 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.503663 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.505247 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.505932 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.506074 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:46.511483 (+ 5409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40223: BlockingRecv error: recv got EOF from 127.0.97.3:40223 (error 108)
Metrics: {"server-negotiator.queue_time_us":420,"spinlock_wait_cycles":6528,"thread_start_us":307,"threads_started":1}
I20260620 02:36:46.512060 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:46.500432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:43645)
0620 02:36:46.500946 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:46.501166 (+ 220us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:46.501348 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:46.501991 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:46.501994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:46.502467 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:46.502699 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:46.502708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.504676 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.504681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:46.506096 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:46.506113 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.507249 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.507258 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:46.507476 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:46.508109 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:46.508127 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:46.511719 (+ 3592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":272,"threads_started":1}
W20260620 02:36:46.512444 9725 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:46.500810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43645 (local address 127.0.97.62:45503)
0620 02:36:46.501636 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:46.501641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:46.501657 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:36:46.501710 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:46.501713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:46.501827 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:46.501933 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:46.502894 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.504499 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.506304 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.507024 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.507129 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:46.512172 (+ 5043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43645: BlockingRecv error: recv got EOF from 127.0.97.1:43645 (error 108)
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":572,"threads_started":1}
Jun 20 02:36:46 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:46 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:46.960361 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:46.951133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:42305)
0620 02:36:46.951959 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:46.951978 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:46.952148 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:46.952529 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:46.952534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:46.952987 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:46.953141 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:46.953149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.954259 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.954265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:46.955216 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:46.955229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.956035 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.956041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:46.956232 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:46.956885 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:46.956905 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:46.960111 (+ 3206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":486,"threads_started":1}
W20260620 02:36:46.961122 9725 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:46.951274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42305 (local address 127.0.97.62:45503)
0620 02:36:46.951489 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:46.951499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:46.952087 (+ 588us) server_negotiation.cc:407] Connection header received
0620 02:36:46.952300 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:46.952305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:46.952379 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:46.952495 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:46.953304 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.954090 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.955386 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:46.955883 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:46.956008 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:46.960889 (+ 4881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42305: BlockingRecv error: recv got EOF from 127.0.97.2:42305 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 02:36:47 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:47 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:47 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:47 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:47.522183 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:47.512583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:58971)
0620 02:36:47.513193 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:47.513225 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:47.513361 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:47.513897 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:47.513902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:47.514294 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:47.514438 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:47.514446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.515459 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.515466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:47.516461 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:47.516473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.517265 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.517272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:47.517463 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:47.518125 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:47.518150 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:47.521805 (+ 3655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":287,"threads_started":1}
I20260620 02:36:47.522626 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:47.513054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:47217)
0620 02:36:47.513575 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:47.513591 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:47.513697 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:47.514239 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:47.514245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:47.514631 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:47.514768 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:47.514777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.515891 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.515896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:47.516791 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:47.516803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.517572 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.517578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:47.517760 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:47.518302 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:47.518318 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:47.522262 (+ 3944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":510,"threads_started":1}
W20260620 02:36:47.522644 9728 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:47.512720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58971 (local address 127.0.97.62:45503)
0620 02:36:47.513463 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:47.513469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:47.513488 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:36:47.513546 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:47.513551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:47.513636 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:47.513839 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:47.514581 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.515318 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.516611 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.517063 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.517233 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:47.522310 (+ 5077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58971: BlockingRecv error: recv got EOF from 127.0.97.3:58971 (error 108)
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":498,"threads_started":1}
W20260620 02:36:47.523262 9730 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:47.513435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47217 (local address 127.0.97.62:45503)
0620 02:36:47.513922 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:47.513926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:47.513940 (+ 14us) server_negotiation.cc:407] Connection header received
0620 02:36:47.513979 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:47.513982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:47.514080 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:47.514201 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:47.514916 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.515714 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.516949 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.517417 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.517548 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:47.523081 (+ 5533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47217: BlockingRecv error: recv got EOF from 127.0.97.1:47217 (error 108)
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":275,"threads_started":1}
Jun 20 02:36:47 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:47 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:47.973421 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:47.961581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:58773)
0620 02:36:47.962247 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:47.962272 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:47.962444 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:47.962909 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:47.962914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:47.963929 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:47.964191 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:47.964205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.965268 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.965276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:47.967144 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:47.967166 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.968248 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.968258 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:47.968473 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:47.969211 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:47.969240 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:47.973016 (+ 3776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":320,"threads_started":1}
W20260620 02:36:47.973462 9730 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:47.961792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58773 (local address 127.0.97.62:45503)
0620 02:36:47.962015 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:47.962021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:47.962358 (+ 337us) server_negotiation.cc:407] Connection header received
0620 02:36:47.962585 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:47.962590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:47.962668 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:47.962767 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:47.964377 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.965097 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.967489 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:47.968031 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:47.968145 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:47.973266 (+ 5121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58773: BlockingRecv error: recv got EOF from 127.0.97.2:58773 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 02:36:48 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:48 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:48 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:48 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:48.534178 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:48.523311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:42563)
0620 02:36:48.523997 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:48.524024 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:48.524195 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:48.524676 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:48.524685 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:48.525290 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:48.525500 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:48.525513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.526612 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.526619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:48.528149 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:48.528161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.529195 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.529199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:48.529327 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:48.529995 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:48.530020 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:48.533787 (+ 3767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":257,"threads_started":1}
W20260620 02:36:48.534226 9733 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:48.523436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42563 (local address 127.0.97.62:45503)
0620 02:36:48.524150 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:48.524155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:48.524176 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:36:48.524348 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:48.524356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:48.524485 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:48.524622 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:48.525643 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.526495 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.528286 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.528952 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.529073 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:48.534019 (+ 4946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42563: BlockingRecv error: recv got EOF from 127.0.97.3:42563 (error 108)
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":520,"threads_started":1}
I20260620 02:36:48.535061 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:48.523494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:37597)
0620 02:36:48.525188 (+ 1694us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:48.525213 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:48.525324 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:48.525783 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:48.525787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:48.526165 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:48.526330 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:48.526340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.527965 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.527969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:48.529384 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:48.529396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.530416 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.530420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:48.530573 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:48.531299 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:48.531321 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:48.534740 (+ 3419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1532,"thread_start_us":1369,"threads_started":1}
W20260620 02:36:48.535171 9735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:48.524102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37597 (local address 127.0.97.62:45503)
0620 02:36:48.525435 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:48.525442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:48.525462 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:36:48.525532 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:48.525536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:48.525609 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:48.525750 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:48.526491 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.527768 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.529611 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.530234 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.530370 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:48.534929 (+ 4559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37597: BlockingRecv error: recv got EOF from 127.0.97.1:37597 (error 108)
Metrics: {"server-negotiator.queue_time_us":1230,"thread_start_us":1054,"threads_started":1}
Jun 20 02:36:48 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:48 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:48.986945 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:48.975271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:40743)
0620 02:36:48.975979 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:48.975997 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:48.976163 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:48.976804 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:48.976808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:48.977397 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:48.977580 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:48.977589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.979334 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.979342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:48.980944 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:48.980984 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.981806 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.981810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:48.981996 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:48.982759 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:48.982787 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:48.986668 (+ 3881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":332,"threads_started":1}
W20260620 02:36:48.987250 9735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:48.975232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40743 (local address 127.0.97.62:45503)
0620 02:36:48.975501 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:48.975509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:48.976127 (+ 618us) server_negotiation.cc:407] Connection header received
0620 02:36:48.976427 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:48.976432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:48.976557 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:48.976721 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:48.977757 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.979017 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.981179 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:48.981638 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:48.981758 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:48.986959 (+ 5201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40743: BlockingRecv error: recv got EOF from 127.0.97.2:40743 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 20 02:36:49 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:49 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:49 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:49 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:49.550766 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:49.535344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.3:43881)
0620 02:36:49.535895 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:49.536116 (+ 221us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:49.536267 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:49.536768 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:49.536772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:49.537278 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:49.537528 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:49.537544 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.538833 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.538837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:49.540044 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:49.540059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.540800 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.540807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:49.540997 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:49.546485 (+ 5488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:49.546507 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:49.550162 (+ 3655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":288,"threads_started":1}
W20260620 02:36:49.550819 9738 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:49.535414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43881 (local address 127.0.97.62:45503)
0620 02:36:49.536384 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:49.536390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:49.536411 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:36:49.536471 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:49.536477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:49.536573 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:49.537506 (+ 933us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:49.537720 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.538626 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.540210 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.540639 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.540756 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:49.550398 (+ 9642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43881: BlockingRecv error: recv got EOF from 127.0.97.3:43881 (error 108)
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":705,"threads_started":1}
I20260620 02:36:49.550917 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:49.536223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.1:60031)
0620 02:36:49.536888 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:49.536908 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:49.537012 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:49.538005 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:49.538008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:49.538357 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:49.538535 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:49.538544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.539602 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.539608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:49.540810 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:49.540826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.541558 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.541564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:49.541737 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:49.546563 (+ 4826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:49.546598 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:49.550550 (+ 3952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":437,"threads_started":1}
W20260620 02:36:49.551316 9740 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:49.536500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60031 (local address 127.0.97.62:45503)
0620 02:36:49.537679 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:49.537685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:49.537700 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:36:49.537735 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:49.537739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:49.537821 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:49.537940 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:49.538678 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.539443 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.540966 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.541390 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.541528 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:49.551016 (+ 9488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60031: BlockingRecv error: recv got EOF from 127.0.97.1:60031 (error 108)
Metrics: {"server-negotiator.queue_time_us":1053,"thread_start_us":885,"threads_started":1}
Jun 20 02:36:49 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:36:49 dist-test-slave-rpg6 krb5kdc[8664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:36:49.997947 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:49.988401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45503 (local address 127.0.97.2:34197)
0620 02:36:49.989058 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:49.989074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:49.989238 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:49.989608 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:49.989613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:49.990074 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:49.990260 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:49.990268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.991604 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.991612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:49.992778 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:49.992790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.993500 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.993506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:49.993664 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:49.994354 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:49.994373 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:49.997560 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":404,"threads_started":1}
W20260620 02:36:49.998019 9740 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:49.988649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34197 (local address 127.0.97.62:45503)
0620 02:36:49.988926 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:49.988934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:49.989249 (+ 315us) server_negotiation.cc:407] Connection header received
0620 02:36:49.989386 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:49.989391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:49.989462 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:49.989587 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:49.990407 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.991469 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.992916 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:49.993347 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:49.993488 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:49.997800 (+ 4312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34197: BlockingRecv error: recv got EOF from 127.0.97.2:34197 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 02:36:50.340152 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 9253
I20260620 02:36:50.355149 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:50.342019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.0.97.1:36527)
0620 02:36:50.342905 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:50.342914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:50.342975 (+ 61us) server_negotiation.cc:407] Connection header received
0620 02:36:50.343038 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:50.343045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:50.343280 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:50.343604 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:50.345254 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:50.346442 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:50.348020 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:50.348627 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:50.348744 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:50.351770 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:50.351780 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:50.351797 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:50.351907 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:50.354580 (+ 2673us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:36:50.354885 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58422: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":569,"threads_started":1}
W20260620 02:36:50.355242 9743 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58422: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:36:50.355485 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 9253: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:36527: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:36:50.355774 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 9253
I20260620 02:36:50.374636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 9404
I20260620 02:36:50.388130 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:50.375929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.0.97.2:37869)
0620 02:36:50.376535 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:50.376542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:50.376574 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:36:50.376636 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:50.376644 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:50.376881 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:50.377049 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:50.378490 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:50.379613 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:50.380993 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:50.381559 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:50.381664 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:50.384444 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:50.384452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:50.384467 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:50.384505 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:50.387275 (+ 2770us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:36:50.387576 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53046: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":290,"threads_started":1}
W20260620 02:36:50.388300 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 9404: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:37869: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:36:50.388377 9744 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53046: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:36:50.388500 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 9404
I20260620 02:36:50.405676 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 9535
I20260620 02:36:50.420316 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:50.406931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.0.97.3:37779)
0620 02:36:50.407561 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:50.407569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:50.407607 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:36:50.407679 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:50.407686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:50.408013 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:50.408236 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:50.409668 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:50.410703 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:50.412061 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:50.412676 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:50.412841 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:50.416215 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:50.416226 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:50.416242 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:50.416294 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:50.419517 (+ 3223us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:36:50.419944 (+ 427us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46788: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":339,"threads_started":1}
W20260620 02:36:50.420459 9745 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46788: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:36:50.420559 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 9535: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:37779: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:36:50.420707 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 9535
I20260620 02:36:50.437844 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 9184
I20260620 02:36:50.439317 9246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:50.559970 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 9184
2026-06-20T02:36:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21673 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T02:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:50Z Disabled control of system clock
I20260620 02:36:50.641037 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32917
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 02:36:51.004245 9750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:51.004572 9750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:51.004717 9750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:51.013091 9750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:51.013221 9750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:51.013263 9750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:51.013381 9750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:51.013463 9750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:36:51.023700 9750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:51.025220 9750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:51.027134 9750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:51.041394 9755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:51.041352 9756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:51.041430 9758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:51.042431 9750 server_base.cc:1034] running on GCE node
I20260620 02:36:51.043344 9750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:51.045358 9750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:51.046586 9750 hybrid_clock.cc:648] HybridClock initialized: now 1781923011046542 us; error 61 us; skew 500 ppm
I20260620 02:36:51.047020 9750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:51.049782 9750 webserver.cc:460] Webserver started at http://127.0.97.62:43245/ using document root <none> and password file <none>
I20260620 02:36:51.050725 9750 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:51.050834 9750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:51.051288 9750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:51.054781 9750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "27ec5379d12448489c2c450826bc7fa5"
format_stamp: "Formatted at 2026-06-20 02:36:51 on dist-test-slave-rpg6"
server_key: "0e7581509ff5333884772999f0e1cb4f"
server_key_iv: "585e70dfe179ab8807a99d9e27906587"
server_key_version: "encryptionkey@0"
I20260620 02:36:51.055774 9750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "27ec5379d12448489c2c450826bc7fa5"
format_stamp: "Formatted at 2026-06-20 02:36:51 on dist-test-slave-rpg6"
server_key: "0e7581509ff5333884772999f0e1cb4f"
server_key_iv: "585e70dfe179ab8807a99d9e27906587"
server_key_version: "encryptionkey@0"
I20260620 02:36:51.062040 9750 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:36:51.066603 9764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:51.068394 9750 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:36:51.068634 9750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "27ec5379d12448489c2c450826bc7fa5"
format_stamp: "Formatted at 2026-06-20 02:36:51 on dist-test-slave-rpg6"
server_key: "0e7581509ff5333884772999f0e1cb4f"
server_key_iv: "585e70dfe179ab8807a99d9e27906587"
server_key_version: "encryptionkey@0"
I20260620 02:36:51.068888 9750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:51.087128 9750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:51.088207 9750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:51.088557 9750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:51.111328 9750 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32917
I20260620 02:36:51.111390 9815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32917 every 8 connection(s)
I20260620 02:36:51.112895 9750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:51.117120 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 9750
I20260620 02:36:51.117368 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:36:51.117580 9816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:51.118171 388 external_mini_cluster.cc:1428] Setting key 245fab7ab5df1912ae5d03b3dacbe165
I20260620 02:36:51.132115 9816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5: Bootstrap starting.
I20260620 02:36:51.133651 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:51.120413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.0.97.62:32917)
0620 02:36:51.121622 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:51.121637 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:51.121682 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:36:51.123925 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:51.123953 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:51.124578 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:51.124945 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:51.125600 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:51.126983 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:51.128962 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:51.129774 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:51.129925 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:51.131183 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:51.131191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:51.131207 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:51.131766 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:51.131929 (+ 163us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:51.131953 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:51.132054 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:51.132415 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 02:36:51.132689 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":499,"threads_started":1}
I20260620 02:36:51.135685 9816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:51.136967 9816 log.cc:824] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:51.140308 9816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5: No bootstrap required, opened a new log
I20260620 02:36:51.148394 9816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ec5379d12448489c2c450826bc7fa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } }
I20260620 02:36:51.148861 9816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:51.148972 9816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ec5379d12448489c2c450826bc7fa5, State: Initialized, Role: FOLLOWER
I20260620 02:36:51.149654 9816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ec5379d12448489c2c450826bc7fa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } }
I20260620 02:36:51.149930 9816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:51.150111 9816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:51.150357 9816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:51.152819 9816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ec5379d12448489c2c450826bc7fa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } }
I20260620 02:36:51.153498 9816 leader_election.cc:302] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27ec5379d12448489c2c450826bc7fa5; no voters:
I20260620 02:36:51.154138 9816 leader_election.cc:288] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:36:51.154327 9821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:51.155057 9821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [term 1 LEADER]: Becoming Leader. State: Replica: 27ec5379d12448489c2c450826bc7fa5, State: Running, Role: LEADER
I20260620 02:36:51.155664 9821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ec5379d12448489c2c450826bc7fa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } }
I20260620 02:36:51.156425 9816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:51.160848 9822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27ec5379d12448489c2c450826bc7fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ec5379d12448489c2c450826bc7fa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } } }
I20260620 02:36:51.160835 9823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27ec5379d12448489c2c450826bc7fa5. Latest consensus state: current_term: 1 leader_uuid: "27ec5379d12448489c2c450826bc7fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ec5379d12448489c2c450826bc7fa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } } }
I20260620 02:36:51.161331 9822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:51.161362 9823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:51.162977 9830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:51.169014 9830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:51.179436 9830 catalog_manager.cc:1347] Generated new cluster ID: dadaae59c01348ea8cc77c5fe1ba883e
I20260620 02:36:51.179596 9830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:51.206791 9830 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:36:51.208348 9830 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:51.214687 9830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27ec5379d12448489c2c450826bc7fa5: Generated new TSK 0
I20260620 02:36:51.215880 9830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:36:51.231050 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32917
--builtin_ntp_servers=127.0.97.20:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 02:36:51.591923 9840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:51.592393 9840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:51.592492 9840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:51.600293 9840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:51.600410 9840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:51.600515 9840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:36:51.610883 9840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:51.612214 9840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:51.613956 9840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:51.628314 9845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:51.628430 9846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:51.628474 9848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:51.629258 9840 server_base.cc:1034] running on GCE node
I20260620 02:36:51.630021 9840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:51.631115 9840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:51.632431 9840 hybrid_clock.cc:648] HybridClock initialized: now 1781923011632358 us; error 64 us; skew 500 ppm
I20260620 02:36:51.632804 9840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:51.635946 9840 webserver.cc:460] Webserver started at http://127.0.97.1:41769/ using document root <none> and password file <none>
I20260620 02:36:51.636777 9840 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:51.636929 9840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:51.637291 9840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:51.640828 9840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "00d6785dada8480499791bd1200a05d8"
format_stamp: "Formatted at 2026-06-20 02:36:51 on dist-test-slave-rpg6"
server_key: "50e77687e42238022f172ad586d18fe0"
server_key_iv: "6b334bb31bddb0085e05d8ea3adf464e"
server_key_version: "encryptionkey@0"
I20260620 02:36:51.641723 9840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "00d6785dada8480499791bd1200a05d8"
format_stamp: "Formatted at 2026-06-20 02:36:51 on dist-test-slave-rpg6"
server_key: "50e77687e42238022f172ad586d18fe0"
server_key_iv: "6b334bb31bddb0085e05d8ea3adf464e"
server_key_version: "encryptionkey@0"
I20260620 02:36:51.647538 9840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:36:51.651983 9854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:51.653582 9840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:36:51.653803 9840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "00d6785dada8480499791bd1200a05d8"
format_stamp: "Formatted at 2026-06-20 02:36:51 on dist-test-slave-rpg6"
server_key: "50e77687e42238022f172ad586d18fe0"
server_key_iv: "6b334bb31bddb0085e05d8ea3adf464e"
server_key_version: "encryptionkey@0"
I20260620 02:36:51.654048 9840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:51.674890 9840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:51.675911 9840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:51.676214 9840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:51.677273 9840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:51.678860 9840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:51.678996 9840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:51.679131 9840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:51.679219 9840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:51.718020 9840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45097
I20260620 02:36:51.718076 9966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45097 every 8 connection(s)
I20260620 02:36:51.719434 9840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:36:51.728463 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 9840
I20260620 02:36:51.728704 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:36:51.729476 388 external_mini_cluster.cc:1428] Setting key 7acd5cadce081228053d00ffacfba5ca
I20260620 02:36:51.732851 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32917
--builtin_ntp_servers=127.0.97.20:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 02:36:51.733851 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:51.722211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59343 (local address 127.0.97.62:32917)
0620 02:36:51.722691 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:51.722697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:51.724188 (+ 1491us) server_negotiation.cc:407] Connection header received
0620 02:36:51.725955 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:51.725960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:51.726030 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:51.726169 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:51.727331 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:51.728169 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:51.730552 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:51.731038 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:51.731151 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:51.732895 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:51.732900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:51.732904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:51.732921 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:51.733011 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:51.733034 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:51.733139 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:51.733394 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 02:36:51.733496 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":227,"threads_started":1}
I20260620 02:36:51.734999 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:51.722852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32917 (local address 127.0.97.1:59343)
0620 02:36:51.723970 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:51.724009 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:51.725655 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:51.726380 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:51.726389 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:51.726481 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:51.727131 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:51.727149 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:51.728365 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:51.728374 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:51.730391 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:51.730407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:51.731304 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:51.731313 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:51.731585 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:51.732482 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:36:51.732488 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:51.732548 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:36:51.732671 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:51.733158 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:51.733170 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:51.733179 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:36:51.733385 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 02:36:51.733864 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":280,"threads_started":1}
I20260620 02:36:51.736883 9967 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32917
I20260620 02:36:51.737218 9967 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:51.738041 9967 heartbeater.cc:511] Master 127.0.97.62:32917 requested a full tablet report, sending...
I20260620 02:36:51.740351 9781 ts_manager.cc:194] Registered new tserver with Master: 00d6785dada8480499791bd1200a05d8 (127.0.97.1:45097)
I20260620 02:36:51.741907 9781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59343
W20260620 02:36:52.078631 9971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:52.079013 9971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:52.079177 9971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:52.086902 9971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:52.087087 9971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:52.087275 9971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:36:52.098655 9971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:52.100632 9971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:52.102641 9971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:52.121021 9977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:52.121083 9976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:52.121793 9979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:52.122145 9971 server_base.cc:1034] running on GCE node
I20260620 02:36:52.122825 9971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:52.123970 9971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:52.125264 9971 hybrid_clock.cc:648] HybridClock initialized: now 1781923012125186 us; error 59 us; skew 500 ppm
I20260620 02:36:52.125645 9971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:52.128229 9971 webserver.cc:460] Webserver started at http://127.0.97.2:44729/ using document root <none> and password file <none>
I20260620 02:36:52.128973 9971 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:52.129112 9971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:52.129446 9971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:52.132469 9971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "35ff456e13c145d2a38638193f10068c"
format_stamp: "Formatted at 2026-06-20 02:36:52 on dist-test-slave-rpg6"
server_key: "767e96f6ff7efc947171bf0b2276babd"
server_key_iv: "cd21a726d831cf0e5728118d815d2403"
server_key_version: "encryptionkey@0"
I20260620 02:36:52.133304 9971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "35ff456e13c145d2a38638193f10068c"
format_stamp: "Formatted at 2026-06-20 02:36:52 on dist-test-slave-rpg6"
server_key: "767e96f6ff7efc947171bf0b2276babd"
server_key_iv: "cd21a726d831cf0e5728118d815d2403"
server_key_version: "encryptionkey@0"
I20260620 02:36:52.139098 9971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:36:52.143060 9985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:52.144789 9971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:36:52.145017 9971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "35ff456e13c145d2a38638193f10068c"
format_stamp: "Formatted at 2026-06-20 02:36:52 on dist-test-slave-rpg6"
server_key: "767e96f6ff7efc947171bf0b2276babd"
server_key_iv: "cd21a726d831cf0e5728118d815d2403"
server_key_version: "encryptionkey@0"
I20260620 02:36:52.145268 9971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:52.165464 9971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:52.166453 9971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:52.166785 9971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:52.167923 9971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:52.169763 9971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:52.169925 9971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:52.170082 9971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:52.170192 9971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:52.207181 9971 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42997
I20260620 02:36:52.207234 10097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42997 every 8 connection(s)
I20260620 02:36:52.208768 9971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:36:52.209270 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 9971
I20260620 02:36:52.209492 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:36:52.210121 388 external_mini_cluster.cc:1428] Setting key 5c54bcdcd554d6be5b5b9521085c9097
I20260620 02:36:52.213176 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32917
--builtin_ntp_servers=127.0.97.20:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 02:36:52.224516 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:52.211750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37457 (local address 127.0.97.62:32917)
0620 02:36:52.211978 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:52.211985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:52.213799 (+ 1814us) server_negotiation.cc:407] Connection header received
0620 02:36:52.215568 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:52.215576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:52.215674 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:52.215795 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:52.217134 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.218337 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.220573 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.220991 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.221152 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:52.223516 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:52.223521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:52.223525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:52.223547 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:52.223582 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:52.223596 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:52.223675 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:52.224166 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 02:36:52.224270 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 02:36:52.226447 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:52.212466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32917 (local address 127.0.97.2:37457)
0620 02:36:52.213614 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:52.213662 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:52.215266 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:52.216122 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:52.216132 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:52.216239 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:52.216903 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:52.216921 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.218750 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.218755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:52.220368 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:52.220389 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.221288 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.221299 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:52.221677 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:52.222655 (+ 978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:36:52.222662 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:52.223067 (+ 405us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:36:52.223212 (+ 145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:52.223807 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:52.223823 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:52.223882 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 02:36:52.224116 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 02:36:52.224803 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":388,"threads_started":1}
I20260620 02:36:52.228410 10098 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32917
I20260620 02:36:52.228803 10098 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:52.229732 10098 heartbeater.cc:511] Master 127.0.97.62:32917 requested a full tablet report, sending...
I20260620 02:36:52.231755 9781 ts_manager.cc:194] Registered new tserver with Master: 35ff456e13c145d2a38638193f10068c (127.0.97.2:42997)
I20260620 02:36:52.232685 9781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:37457
W20260620 02:36:52.570078 10101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:52.570449 10101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:52.570600 10101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:52.578537 10101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:52.578678 10101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:52.578868 10101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:36:52.588478 10101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:52.589891 10101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:52.591883 10101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:52.606616 10107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:52.606699 10106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:52.606699 10109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:52.608163 10101 server_base.cc:1034] running on GCE node
I20260620 02:36:52.608923 10101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:52.610013 10101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:52.611330 10101 hybrid_clock.cc:648] HybridClock initialized: now 1781923012611266 us; error 46 us; skew 500 ppm
I20260620 02:36:52.611745 10101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:52.614641 10101 webserver.cc:460] Webserver started at http://127.0.97.3:44717/ using document root <none> and password file <none>
I20260620 02:36:52.615568 10101 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:52.615733 10101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:52.616149 10101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:52.619351 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "a64a3838503649adac011260036b013a"
format_stamp: "Formatted at 2026-06-20 02:36:52 on dist-test-slave-rpg6"
server_key: "9d367332cbae7abbb272fe33000ed112"
server_key_iv: "7a582d469483a5cbc81daaada3b2ec04"
server_key_version: "encryptionkey@0"
I20260620 02:36:52.620347 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a64a3838503649adac011260036b013a"
format_stamp: "Formatted at 2026-06-20 02:36:52 on dist-test-slave-rpg6"
server_key: "9d367332cbae7abbb272fe33000ed112"
server_key_iv: "7a582d469483a5cbc81daaada3b2ec04"
server_key_version: "encryptionkey@0"
I20260620 02:36:52.626420 10101 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:36:52.630537 10115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:52.632463 10101 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:36:52.632696 10101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "a64a3838503649adac011260036b013a"
format_stamp: "Formatted at 2026-06-20 02:36:52 on dist-test-slave-rpg6"
server_key: "9d367332cbae7abbb272fe33000ed112"
server_key_iv: "7a582d469483a5cbc81daaada3b2ec04"
server_key_version: "encryptionkey@0"
I20260620 02:36:52.632968 10101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:52.656711 10101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:52.657699 10101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:52.657994 10101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:52.659116 10101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:52.660822 10101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:52.660955 10101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:52.661090 10101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:52.661197 10101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:52.697293 10101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44955
I20260620 02:36:52.697538 10227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44955 every 8 connection(s)
I20260620 02:36:52.698832 10101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:36:52.703411 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 10101
I20260620 02:36:52.703768 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:36:52.704416 388 external_mini_cluster.cc:1428] Setting key b71c5918e18450919858d4192a24fb38
I20260620 02:36:52.714527 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:52.701797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51259 (local address 127.0.97.62:32917)
0620 02:36:52.701992 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:52.701997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:52.704091 (+ 2094us) server_negotiation.cc:407] Connection header received
0620 02:36:52.706420 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:52.706424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:52.706547 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:52.706671 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:52.708433 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.709220 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.711077 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.711538 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.711650 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:52.713482 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:52.713491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:52.713495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:52.713538 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:52.713597 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:52.713615 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:52.713949 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:52.714126 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 02:36:52.714237 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:36:52.715574 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:52.702363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32917 (local address 127.0.97.3:51259)
0620 02:36:52.703881 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:52.703937 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:52.706007 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:52.707260 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:52.707272 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:52.707382 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:52.708229 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:52.708253 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.709395 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.709399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:52.710868 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:52.710886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.711821 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.711888 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:52.712154 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:52.712914 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:36:52.712917 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:52.712962 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:36:52.713061 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:52.713784 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:52.713794 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:52.713802 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 02:36:52.713982 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 02:36:52.714367 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":337,"threads_started":1}
I20260620 02:36:52.717675 10228 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32917
I20260620 02:36:52.718067 10228 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:52.719022 10228 heartbeater.cc:511] Master 127.0.97.62:32917 requested a full tablet report, sending...
I20260620 02:36:52.720811 9780 ts_manager.cc:194] Registered new tserver with Master: a64a3838503649adac011260036b013a (127.0.97.3:44955)
I20260620 02:36:52.721712 9780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:51259
I20260620 02:36:52.733721 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 02:36:52.741423 9969 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:52.739458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.0.97.62:32917)
0620 02:36:52.739726 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:52.739731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:52.740204 (+ 473us) server_negotiation.cc:407] Connection header received
0620 02:36:52.740355 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:52.740360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:52.740432 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:52.740538 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:52.741180 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41476: BlockingRecv error: recv got EOF from 127.0.0.1:41476 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 02:36:52.741909 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 9840
I20260620 02:36:52.745409 9967 heartbeater.cc:503] Master 127.0.97.62:32917 was elected leader, sending a full tablet report...
I20260620 02:36:52.752106 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:52.743468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54044 (local address 127.0.97.1:45097)
0620 02:36:52.744017 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:52.744023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:52.744051 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:36:52.744109 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:52.744115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:52.744348 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:52.744487 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:52.744882 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.746635 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.748136 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.748817 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.748977 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:52.750490 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:52.750507 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:52.750524 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:52.751143 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:52.751196 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:52.751225 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:52.751366 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:52.751537 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 02:36:52.751680 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":253,"threads_started":1}
I20260620 02:36:52.753600 9962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:52.839000 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 9840
I20260620 02:36:52.856092 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 9971
I20260620 02:36:52.864364 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:52.857074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.0.97.2:42997)
0620 02:36:52.857602 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:52.857609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:52.857635 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:36:52.857698 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:52.857704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:52.857920 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:52.858102 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:52.858502 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.859473 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.860794 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.861388 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.861530 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:52.862669 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:52.862678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:52.862693 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:52.863447 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:52.863498 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:52.863518 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:52.863653 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:52.863814 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 02:36:52.863997 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":290,"threads_started":1}
I20260620 02:36:52.865779 10093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:52.949502 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 9971
I20260620 02:36:52.967031 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 10101
I20260620 02:36:52.977298 10241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:52.968356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.0.97.3:44955)
0620 02:36:52.969176 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:52.969183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:52.969216 (+ 33us) server_negotiation.cc:407] Connection header received
0620 02:36:52.969287 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:52.969294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:52.969530 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:52.969807 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:52.970354 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.971414 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.973234 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:52.973862 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:52.974017 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:52.975467 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:52.975478 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:52.975496 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:52.976291 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:52.976345 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:52.976370 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:52.976524 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:52.976727 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 02:36:52.976876 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":515,"threads_started":1}
I20260620 02:36:52.978749 10223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:53.065285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 10101
I20260620 02:36:53.081940 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 9750
I20260620 02:36:53.083411 9811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:53.188297 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 9750
2026-06-20T02:36:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2619 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T02:36:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:53Z Disabled control of system clock
I20260620 02:36:53.260520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32825
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 02:36:53.626727 10247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:53.627081 10247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:53.627175 10247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:53.634788 10247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:53.634909 10247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:53.634946 10247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:53.634995 10247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:53.635039 10247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:36:53.644588 10247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:53.645917 10247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:53.647768 10247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:53.660529 10255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:53.660570 10252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:53.660548 10253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:53.661267 10247 server_base.cc:1034] running on GCE node
I20260620 02:36:53.662096 10247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:53.663573 10247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:53.664832 10247 hybrid_clock.cc:648] HybridClock initialized: now 1781923013664782 us; error 43 us; skew 500 ppm
I20260620 02:36:53.665246 10247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:53.667773 10247 webserver.cc:460] Webserver started at http://127.0.97.62:39351/ using document root <none> and password file <none>
I20260620 02:36:53.668605 10247 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:53.668741 10247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:53.669085 10247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:53.674305 10247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "bfa10401f41f4913b8a275c73b4a5ffc"
format_stamp: "Formatted at 2026-06-20 02:36:53 on dist-test-slave-rpg6"
server_key: "062b18b7e06d5a697fac553992091ab6"
server_key_iv: "8bbb43a8d2d921d56246d04642d10ce1"
server_key_version: "encryptionkey@0"
I20260620 02:36:53.675314 10247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "bfa10401f41f4913b8a275c73b4a5ffc"
format_stamp: "Formatted at 2026-06-20 02:36:53 on dist-test-slave-rpg6"
server_key: "062b18b7e06d5a697fac553992091ab6"
server_key_iv: "8bbb43a8d2d921d56246d04642d10ce1"
server_key_version: "encryptionkey@0"
I20260620 02:36:53.681243 10247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:36:53.685420 10261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:53.686972 10247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 02:36:53.687196 10247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "bfa10401f41f4913b8a275c73b4a5ffc"
format_stamp: "Formatted at 2026-06-20 02:36:53 on dist-test-slave-rpg6"
server_key: "062b18b7e06d5a697fac553992091ab6"
server_key_iv: "8bbb43a8d2d921d56246d04642d10ce1"
server_key_version: "encryptionkey@0"
I20260620 02:36:53.687484 10247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:53.694084 10247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:53.694994 10247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:53.695351 10247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:53.717068 10247 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32825
I20260620 02:36:53.717090 10312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32825 every 8 connection(s)
I20260620 02:36:53.718585 10247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:53.722738 10313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:53.726727 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 10247
I20260620 02:36:53.726981 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:36:53.727803 388 external_mini_cluster.cc:1428] Setting key 2c01329dca47704355867f13b823309c
I20260620 02:36:53.737326 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:53.730173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.0.97.62:32825)
0620 02:36:53.731356 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:53.731368 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:53.731413 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:36:53.733200 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:53.733228 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:53.733764 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:53.734260 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:53.735487 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:53.735494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:53.735508 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:53.735523 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:53.735593 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:53.735611 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:53.735890 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:53.736036 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 02:36:53.736333 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":398,"threads_started":1}
I20260620 02:36:53.737475 10313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc: Bootstrap starting.
I20260620 02:36:53.741433 10313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:53.742565 10313 log.cc:824] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:53.746567 10313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc: No bootstrap required, opened a new log
I20260620 02:36:53.756486 10313 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa10401f41f4913b8a275c73b4a5ffc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32825 } }
I20260620 02:36:53.756925 10313 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:53.757025 10313 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa10401f41f4913b8a275c73b4a5ffc, State: Initialized, Role: FOLLOWER
I20260620 02:36:53.757767 10313 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa10401f41f4913b8a275c73b4a5ffc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32825 } }
I20260620 02:36:53.758023 10313 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:53.758160 10313 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:53.758337 10313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:53.760831 10313 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa10401f41f4913b8a275c73b4a5ffc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32825 } }
I20260620 02:36:53.761507 10313 leader_election.cc:302] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfa10401f41f4913b8a275c73b4a5ffc; no voters:
I20260620 02:36:53.762343 10313 leader_election.cc:288] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:36:53.762483 10318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:53.763430 10318 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [term 1 LEADER]: Becoming Leader. State: Replica: bfa10401f41f4913b8a275c73b4a5ffc, State: Running, Role: LEADER
I20260620 02:36:53.764158 10318 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa10401f41f4913b8a275c73b4a5ffc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32825 } }
I20260620 02:36:53.765312 10313 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:53.767686 10319 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfa10401f41f4913b8a275c73b4a5ffc. Latest consensus state: current_term: 1 leader_uuid: "bfa10401f41f4913b8a275c73b4a5ffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa10401f41f4913b8a275c73b4a5ffc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32825 } } }
I20260620 02:36:53.767664 10320 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfa10401f41f4913b8a275c73b4a5ffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfa10401f41f4913b8a275c73b4a5ffc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32825 } } }
I20260620 02:36:53.768849 10320 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:53.770851 10319 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:53.771586 10327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:53.778543 10327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:53.789615 10327 catalog_manager.cc:1347] Generated new cluster ID: ac677bc9620242e290bc21f024dae8e5
I20260620 02:36:53.789947 10327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:53.821322 10327 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:36:53.821856 10327 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:53.834321 10327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfa10401f41f4913b8a275c73b4a5ffc: Generated new TSK 0
I20260620 02:36:53.835317 10327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:36:53.843357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32825
--builtin_ntp_servers=127.0.97.20:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 02:36:54.225028 10337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:54.225368 10337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:54.225466 10337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:54.234310 10337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:54.234549 10337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:54.234932 10337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:36:54.246371 10337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:54.247991 10337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:54.249938 10337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:54.265725 10342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:54.265690 10343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:54.265621 10345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:54.267474 10337 server_base.cc:1034] running on GCE node
I20260620 02:36:54.268291 10337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:54.269605 10337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:54.271103 10337 hybrid_clock.cc:648] HybridClock initialized: now 1781923014271037 us; error 63 us; skew 500 ppm
I20260620 02:36:54.271549 10337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:54.274878 10337 webserver.cc:460] Webserver started at http://127.0.97.1:33277/ using document root <none> and password file <none>
I20260620 02:36:54.275619 10337 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:54.275730 10337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:54.276098 10337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:54.279268 10337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "a6bc04f24b52452589f9a61845af4492"
format_stamp: "Formatted at 2026-06-20 02:36:54 on dist-test-slave-rpg6"
server_key: "36903453b63c1464268ca0ec30219864"
server_key_iv: "9e9a24e3577306cf4a0fe76979adebdd"
server_key_version: "encryptionkey@0"
I20260620 02:36:54.280223 10337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a6bc04f24b52452589f9a61845af4492"
format_stamp: "Formatted at 2026-06-20 02:36:54 on dist-test-slave-rpg6"
server_key: "36903453b63c1464268ca0ec30219864"
server_key_iv: "9e9a24e3577306cf4a0fe76979adebdd"
server_key_version: "encryptionkey@0"
I20260620 02:36:54.286180 10337 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:36:54.291004 10351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:54.292997 10337 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:36:54.293262 10337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "a6bc04f24b52452589f9a61845af4492"
format_stamp: "Formatted at 2026-06-20 02:36:54 on dist-test-slave-rpg6"
server_key: "36903453b63c1464268ca0ec30219864"
server_key_iv: "9e9a24e3577306cf4a0fe76979adebdd"
server_key_version: "encryptionkey@0"
I20260620 02:36:54.293613 10337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:54.302356 10337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:54.303664 10337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:54.304028 10337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:54.305394 10337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:54.307013 10337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:54.307132 10337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:54.307221 10337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:54.307269 10337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:54.349345 10337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40289
I20260620 02:36:54.349382 10463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40289 every 8 connection(s)
I20260620 02:36:54.350824 10337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:36:54.351423 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 10337
I20260620 02:36:54.351691 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:36:54.353221 388 external_mini_cluster.cc:1428] Setting key 1cba1e799c163e4e0ca68ac61a0bb24e
I20260620 02:36:54.357046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32825
--builtin_ntp_servers=127.0.97.20:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 02:36:54.362953 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:54.354117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56867 (local address 127.0.97.62:32825)
0620 02:36:54.354689 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:54.354695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:54.356188 (+ 1493us) server_negotiation.cc:407] Connection header received
0620 02:36:54.359519 (+ 3331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:54.359527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:54.359616 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:54.359749 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:54.361967 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:54.361976 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:54.361984 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:54.362014 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:54.362074 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:54.362096 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:54.362250 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:54.362461 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 02:36:54.362534 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":280,"threads_started":1}
I20260620 02:36:54.364274 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:54.354771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32825 (local address 127.0.97.1:56867)
0620 02:36:54.355979 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:54.356026 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:54.358841 (+ 2815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:54.360267 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:54.360283 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:54.360431 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:54.361529 (+ 1098us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:36:54.361535 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:54.361591 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:36:54.361713 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:54.362211 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:54.362222 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:54.362233 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:36:54.362521 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 02:36:54.362808 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":372,"threads_started":1}
I20260620 02:36:54.366797 10464 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32825
I20260620 02:36:54.367268 10464 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:54.368197 10464 heartbeater.cc:511] Master 127.0.97.62:32825 requested a full tablet report, sending...
I20260620 02:36:54.370831 10278 ts_manager.cc:194] Registered new tserver with Master: a6bc04f24b52452589f9a61845af4492 (127.0.97.1:40289)
W20260620 02:36:54.730412 10468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:54.730721 10468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:54.730904 10468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:54.738612 10468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:54.738766 10468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:54.738948 10468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:36:54.748737 10468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:54.750141 10468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:54.752053 10468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:54.766948 10473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:54.767204 10474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:54.767230 10476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:54.768664 10468 server_base.cc:1034] running on GCE node
I20260620 02:36:54.769369 10468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:54.770624 10468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:54.772002 10468 hybrid_clock.cc:648] HybridClock initialized: now 1781923014771954 us; error 77 us; skew 500 ppm
I20260620 02:36:54.772552 10468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:54.780579 10468 webserver.cc:460] Webserver started at http://127.0.97.2:33837/ using document root <none> and password file <none>
I20260620 02:36:54.781424 10468 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:54.781546 10468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:54.781828 10468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:54.785053 10468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "d475ae9218034b358ee2fa36937cf207"
format_stamp: "Formatted at 2026-06-20 02:36:54 on dist-test-slave-rpg6"
server_key: "776bb6616c52d61737478a9f851b12e0"
server_key_iv: "18c29139f871d27020b9ca73eb135386"
server_key_version: "encryptionkey@0"
I20260620 02:36:54.785943 10468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d475ae9218034b358ee2fa36937cf207"
format_stamp: "Formatted at 2026-06-20 02:36:54 on dist-test-slave-rpg6"
server_key: "776bb6616c52d61737478a9f851b12e0"
server_key_iv: "18c29139f871d27020b9ca73eb135386"
server_key_version: "encryptionkey@0"
I20260620 02:36:54.792423 10468 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 02:36:54.796598 10482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:54.798355 10468 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:36:54.798518 10468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "d475ae9218034b358ee2fa36937cf207"
format_stamp: "Formatted at 2026-06-20 02:36:54 on dist-test-slave-rpg6"
server_key: "776bb6616c52d61737478a9f851b12e0"
server_key_iv: "18c29139f871d27020b9ca73eb135386"
server_key_version: "encryptionkey@0"
I20260620 02:36:54.798835 10468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:54.805424 10468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:54.806388 10468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:54.806720 10468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:54.807801 10468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:54.809420 10468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:54.809512 10468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:54.809659 10468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:54.809744 10468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:54.849550 10468 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38861
I20260620 02:36:54.849665 10594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38861 every 8 connection(s)
I20260620 02:36:54.851322 10468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:36:54.858309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 10468
I20260620 02:36:54.858456 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:36:54.859040 388 external_mini_cluster.cc:1428] Setting key 5d419c4b4678fc3d1d6da0b5af3138ca
I20260620 02:36:54.861750 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:54.854351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60941 (local address 127.0.97.62:32825)
0620 02:36:54.854622 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:54.854627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:54.856272 (+ 1645us) server_negotiation.cc:407] Connection header received
0620 02:36:54.858290 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:54.858295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:54.858357 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:54.858522 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:54.860291 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:54.860300 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:54.860306 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:54.860331 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:54.860385 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:54.860404 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:54.860552 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:54.861417 (+ 865us) server_negotiation.cc:299] Negotiation successful
0620 02:36:54.861489 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 02:36:54.862509 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32825
--builtin_ntp_servers=127.0.97.20:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 02:36:54.862988 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:54.854819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32825 (local address 127.0.97.2:60941)
0620 02:36:54.856046 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:54.856130 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:54.857756 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:54.858862 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:54.858876 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:54.859024 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:54.859910 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:36:54.859913 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:54.859957 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:36:54.860058 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:54.861161 (+ 1103us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:54.861174 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:54.861183 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:36:54.861544 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 02:36:54.861876 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":303,"threads_started":1}
I20260620 02:36:54.865561 10595 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32825
I20260620 02:36:54.865945 10595 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:54.866851 10595 heartbeater.cc:511] Master 127.0.97.62:32825 requested a full tablet report, sending...
I20260620 02:36:54.869006 10278 ts_manager.cc:194] Registered new tserver with Master: d475ae9218034b358ee2fa36937cf207 (127.0.97.2:38861)
W20260620 02:36:55.237614 10598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:55.237954 10598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:55.238118 10598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:55.246013 10598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:55.246150 10598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:55.246361 10598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:36:55.256574 10598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:55.258095 10598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:55.260042 10598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:55.276700 10606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:55.276767 10603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:55.276729 10604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:55.277906 10598 server_base.cc:1034] running on GCE node
I20260620 02:36:55.278954 10598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:55.280540 10598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:55.281862 10598 hybrid_clock.cc:648] HybridClock initialized: now 1781923015281807 us; error 87 us; skew 500 ppm
I20260620 02:36:55.282612 10598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:36:55.286796 10598 webserver.cc:460] Webserver started at http://127.0.97.3:37699/ using document root <none> and password file <none>
I20260620 02:36:55.287951 10598 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:55.288161 10598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:55.288561 10598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:55.291975 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "838bf79290094fb9a64a11c8364ab548"
format_stamp: "Formatted at 2026-06-20 02:36:55 on dist-test-slave-rpg6"
server_key: "1ba0b79a71b61e9d597ef7125e9b56be"
server_key_iv: "8e46f576c7cf3ac589fd096e0ce4cdd4"
server_key_version: "encryptionkey@0"
I20260620 02:36:55.293015 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "838bf79290094fb9a64a11c8364ab548"
format_stamp: "Formatted at 2026-06-20 02:36:55 on dist-test-slave-rpg6"
server_key: "1ba0b79a71b61e9d597ef7125e9b56be"
server_key_iv: "8e46f576c7cf3ac589fd096e0ce4cdd4"
server_key_version: "encryptionkey@0"
I20260620 02:36:55.299389 10598 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:36:55.304263 10612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:55.306720 10598 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:36:55.307056 10598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "838bf79290094fb9a64a11c8364ab548"
format_stamp: "Formatted at 2026-06-20 02:36:55 on dist-test-slave-rpg6"
server_key: "1ba0b79a71b61e9d597ef7125e9b56be"
server_key_iv: "8e46f576c7cf3ac589fd096e0ce4cdd4"
server_key_version: "encryptionkey@0"
I20260620 02:36:55.307386 10598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:55.315464 10598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:55.316527 10598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:55.316847 10598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:55.317914 10598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:55.319448 10598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:55.319540 10598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:55.319696 10598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:55.319778 10598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:55.361234 10598 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44101
I20260620 02:36:55.361258 10724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44101 every 8 connection(s)
I20260620 02:36:55.362668 10598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:36:55.371628 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 10598
I20260620 02:36:55.371847 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:36:55.372227 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:55.365613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48897 (local address 127.0.97.62:32825)
0620 02:36:55.366131 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:55.366138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:55.367495 (+ 1357us) server_negotiation.cc:407] Connection header received
0620 02:36:55.369435 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:55.369441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:55.369515 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:55.369656 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:55.371181 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:55.371188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:55.371193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:55.371228 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:55.371280 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:55.371295 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:55.371445 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:55.371701 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 02:36:55.371780 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":306,"threads_started":1}
I20260620 02:36:55.372378 388 external_mini_cluster.cc:1428] Setting key 318a9db05b9c34b77354dd3874b17c94
I20260620 02:36:55.373586 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:55.366251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32825 (local address 127.0.97.3:48897)
0620 02:36:55.367322 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:55.367361 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:55.369128 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:55.369917 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:55.369926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:55.370036 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:55.370799 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:36:55.370802 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:55.370846 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:36:55.370945 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:55.371411 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:55.371422 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:55.371431 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:36:55.371800 (+ 369us) client_negotiation.cc:240] Negotiation successful
0620 02:36:55.372272 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":296,"threads_started":1}
I20260620 02:36:55.375262 10725 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32825
I20260620 02:36:55.375313 10464 heartbeater.cc:503] Master 127.0.97.62:32825 was elected leader, sending a full tablet report...
I20260620 02:36:55.375555 10725 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:55.376329 10725 heartbeater.cc:511] Master 127.0.97.62:32825 requested a full tablet report, sending...
I20260620 02:36:55.377918 10277 ts_manager.cc:194] Registered new tserver with Master: 838bf79290094fb9a64a11c8364ab548 (127.0.97.3:44101)
I20260620 02:36:55.388898 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 02:36:55.396338 10727 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:36:55.394272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.0.97.62:32825)
0620 02:36:55.394478 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:55.394483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:55.395127 (+ 644us) server_negotiation.cc:407] Connection header received
0620 02:36:55.395257 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:55.395263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:55.395324 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:55.395422 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:55.395991 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44706: BlockingRecv error: recv got EOF from 127.0.0.1:44706 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 02:36:55.396872 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 10337
I20260620 02:36:55.401216 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:55.397918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.0.97.1:40289)
0620 02:36:55.398698 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:55.398705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:55.398734 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:36:55.398794 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:55.398846 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:55.399084 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:55.399337 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:55.400433 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:55.400439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:55.400451 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:55.400463 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:55.400502 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:55.400516 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:55.400660 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:55.400844 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 02:36:55.400931 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":483,"threads_started":1}
I20260620 02:36:55.402583 10459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:55.488433 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 10337
I20260620 02:36:55.505527 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 10468
I20260620 02:36:55.509621 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:55.506632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.0.97.2:38861)
0620 02:36:55.507167 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:55.507175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:55.507203 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:36:55.507277 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:55.507286 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:55.507504 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:55.507634 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:55.508726 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:55.508736 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:55.508748 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:55.508764 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:55.508816 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:55.508835 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:55.508959 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:55.509113 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 02:36:55.509379 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":270,"threads_started":1}
I20260620 02:36:55.511047 10590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:55.598680 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 10468
I20260620 02:36:55.616796 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 10598
I20260620 02:36:55.621768 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:55.618345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.0.97.3:44101)
0620 02:36:55.619007 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:55.619022 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:55.619056 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:36:55.619118 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:55.619125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:55.619426 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:55.619638 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:55.620962 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:55.620969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:55.620981 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:36:55.620994 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:55.621032 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:36:55.621046 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:55.621272 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:55.621348 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 02:36:55.621463 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":346,"threads_started":1}
I20260620 02:36:55.623168 10720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:55.709892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 10598
I20260620 02:36:55.730063 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 10247
I20260620 02:36:55.731462 10308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:36:55.837113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 10247
2026-06-20T02:36:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2652 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:36:55 dist-test-slave-rpg6 krb5kdc[10744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:36:55 dist-test-slave-rpg6 krb5kdc[10744](info): set up 2 sockets
Jun 20 02:36:55 dist-test-slave-rpg6 krb5kdc[10744](info): commencing operation
krb5kdc: starting...
W20260620 02:36:56.990171 388 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:36:57 dist-test-slave-rpg6 krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:36:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:36:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:36:57.269937 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34177
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:57.625298 10757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:57.625618 10757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:57.625784 10757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:57.633360 10757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:36:57.633498 10757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:57.633545 10757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:57.633675 10757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:36:57.633725 10757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:36:57.644146 10757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:57.645622 10757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:57.647043 10757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:57.659940 10763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:57.660066 10765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:57.659981 10762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:57.661046 10757 server_base.cc:1034] running on GCE node
I20260620 02:36:57.661991 10757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:57.663539 10757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:57.664814 10757 hybrid_clock.cc:648] HybridClock initialized: now 1781923017664759 us; error 52 us; skew 500 ppm
I20260620 02:36:57.665210 10757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:57 dist-test-slave-rpg6 krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:57.668655 10757 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:36:57.670409 10757 webserver.cc:460] Webserver started at http://127.0.97.62:34363/ using document root <none> and password file <none>
I20260620 02:36:57.671111 10757 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:57.671217 10757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:57.671496 10757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:57.674885 10757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "ca2274b637aa44439a31a2900af8a093"
format_stamp: "Formatted at 2026-06-20 02:36:57 on dist-test-slave-rpg6"
server_key: "37ed0b140de9babe57ddbafdf41b71cc"
server_key_iv: "e7deebdda194353781ce9368e69f358a"
server_key_version: "encryptionkey@0"
I20260620 02:36:57.675693 10757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "ca2274b637aa44439a31a2900af8a093"
format_stamp: "Formatted at 2026-06-20 02:36:57 on dist-test-slave-rpg6"
server_key: "37ed0b140de9babe57ddbafdf41b71cc"
server_key_iv: "e7deebdda194353781ce9368e69f358a"
server_key_version: "encryptionkey@0"
I20260620 02:36:57.681661 10757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:36:57.685869 10772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:57.687618 10757 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 02:36:57.687915 10757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "ca2274b637aa44439a31a2900af8a093"
format_stamp: "Formatted at 2026-06-20 02:36:57 on dist-test-slave-rpg6"
server_key: "37ed0b140de9babe57ddbafdf41b71cc"
server_key_iv: "e7deebdda194353781ce9368e69f358a"
server_key_version: "encryptionkey@0"
I20260620 02:36:57.688172 10757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:57.702770 10757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:57.707536 10757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:57.707896 10757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:57.730296 10757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34177
I20260620 02:36:57.730365 10823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34177 every 8 connection(s)
I20260620 02:36:57.731946 10757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:36:57.736310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 10757
I20260620 02:36:57.736548 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:36:57.736690 10824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:57.737673 388 external_mini_cluster.cc:1428] Setting key 1dc7213e27c390947df790d7de315be6
I20260620 02:36:57.751142 10824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093: Bootstrap starting.
Jun 20 02:36:57 dist-test-slave-rpg6 krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:36:57.754680 10824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:57.756095 10824 log.cc:824] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:57.759670 10824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093: No bootstrap required, opened a new log
I20260620 02:36:57.763657 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:57.740172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.0.97.62:34177)
0620 02:36:57.741074 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:57.741093 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:57.741137 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:36:57.743324 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:57.743355 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:57.743900 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:57.744567 (+ 667us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:57.746181 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:57.747596 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:57.749029 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:57.749795 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:57.750016 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:57.754624 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:57.754634 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:57.754652 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:57.754705 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:57.758687 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:57.759919 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:57.759928 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:57.759936 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:57.760131 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:57.760601 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:57.760605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:57.760607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:57.761194 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:57.761576 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:57.762023 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 02:36:57.762509 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":666,"thread_start_us":425,"threads_started":1}
I20260620 02:36:57.769048 10824 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2274b637aa44439a31a2900af8a093" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34177 } }
I20260620 02:36:57.769647 10824 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:57.769788 10824 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2274b637aa44439a31a2900af8a093, State: Initialized, Role: FOLLOWER
I20260620 02:36:57.770509 10824 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2274b637aa44439a31a2900af8a093" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34177 } }
I20260620 02:36:57.770787 10824 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:36:57.770964 10824 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:36:57.771382 10824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:57.774401 10824 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2274b637aa44439a31a2900af8a093" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34177 } }
I20260620 02:36:57.774945 10824 leader_election.cc:302] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca2274b637aa44439a31a2900af8a093; no voters:
I20260620 02:36:57.775750 10824 leader_election.cc:288] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:36:57.776094 10829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:57.777566 10829 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [term 1 LEADER]: Becoming Leader. State: Replica: ca2274b637aa44439a31a2900af8a093, State: Running, Role: LEADER
I20260620 02:36:57.778335 10829 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2274b637aa44439a31a2900af8a093" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34177 } }
I20260620 02:36:57.778829 10824 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:36:57.781677 10830 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca2274b637aa44439a31a2900af8a093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2274b637aa44439a31a2900af8a093" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34177 } } }
I20260620 02:36:57.782249 10830 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:57.781832 10831 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca2274b637aa44439a31a2900af8a093. Latest consensus state: current_term: 1 leader_uuid: "ca2274b637aa44439a31a2900af8a093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2274b637aa44439a31a2900af8a093" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34177 } } }
I20260620 02:36:57.782370 10831 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093 [sys.catalog]: This master's current role is: LEADER
I20260620 02:36:57.790722 10837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:36:57.796167 10837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:36:57.805610 10837 catalog_manager.cc:1347] Generated new cluster ID: 0150a8ea852c42ae8a4c92e9b23545a0
I20260620 02:36:57.805757 10837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:36:57.823668 10837 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:36:57.825703 10837 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:36:57.841809 10837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093: Generated new TSK 0
I20260620 02:36:57.842880 10837 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:36:57.913887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34177
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:58.275674 10850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:58.276113 10850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:58.276218 10850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:58.285053 10850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:58.285193 10850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:58.285338 10850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:36:58.300807 10850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:58.302201 10850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:58.303519 10850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:58.318388 10858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:58.318413 10855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:58.318529 10856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:58.319272 10850 server_base.cc:1034] running on GCE node
I20260620 02:36:58.320051 10850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:58.321143 10850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:58.322490 10850 hybrid_clock.cc:648] HybridClock initialized: now 1781923018322410 us; error 49 us; skew 500 ppm
I20260620 02:36:58.323369 10850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:58 dist-test-slave-rpg6 krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:58.327081 10850 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:36:58.329087 10850 webserver.cc:460] Webserver started at http://127.0.97.1:46417/ using document root <none> and password file <none>
I20260620 02:36:58.329902 10850 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:58.330058 10850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:58.330379 10850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:58.333607 10850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "acc93d8f809e40d8ac421432067a11d1"
format_stamp: "Formatted at 2026-06-20 02:36:58 on dist-test-slave-rpg6"
server_key: "b4420addf738de21cd60e2a3b87f0ff9"
server_key_iv: "cd5c80960682fd632b7d0866f8f961f6"
server_key_version: "encryptionkey@0"
I20260620 02:36:58.334473 10850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "acc93d8f809e40d8ac421432067a11d1"
format_stamp: "Formatted at 2026-06-20 02:36:58 on dist-test-slave-rpg6"
server_key: "b4420addf738de21cd60e2a3b87f0ff9"
server_key_iv: "cd5c80960682fd632b7d0866f8f961f6"
server_key_version: "encryptionkey@0"
I20260620 02:36:58.340415 10850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 02:36:58.344606 10865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:58.346537 10850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:36:58.346800 10850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "acc93d8f809e40d8ac421432067a11d1"
format_stamp: "Formatted at 2026-06-20 02:36:58 on dist-test-slave-rpg6"
server_key: "b4420addf738de21cd60e2a3b87f0ff9"
server_key_iv: "cd5c80960682fd632b7d0866f8f961f6"
server_key_version: "encryptionkey@0"
I20260620 02:36:58.347038 10850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:58.362634 10850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:58.367568 10850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:58.368008 10850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:58.369087 10850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:58.370752 10850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:58.370872 10850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:58.371031 10850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:58.371131 10850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:58.409649 10850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34073
I20260620 02:36:58.409657 10977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34073 every 8 connection(s)
I20260620 02:36:58.411039 10850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:36:58.421132 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 10850
I20260620 02:36:58.421356 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:36:58.422196 388 external_mini_cluster.cc:1428] Setting key 9e6820f7dd12f40be74ac889925525d3
Jun 20 02:36:58 dist-test-slave-rpg6 krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:36:58.437177 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:58.414918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59533 (local address 127.0.97.62:34177)
0620 02:36:58.415472 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:58.415477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:58.416606 (+ 1129us) server_negotiation.cc:407] Connection header received
0620 02:36:58.418537 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:58.418542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:58.418615 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:58.418754 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:58.420545 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:58.421488 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:58.423810 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:58.424497 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:58.424673 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:58.429911 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:58.429917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:58.429920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:58.429957 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:58.433320 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:58.434497 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:58.434502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:58.434504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:58.434642 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:58.435201 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:58.435205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:58.435208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:58.435501 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:58.435660 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:58.436702 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0620 02:36:58.436888 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":313,"threads_started":1}
I20260620 02:36:58.438666 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:58.415517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34177 (local address 127.0.97.1:59533)
0620 02:36:58.416412 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:58.416453 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:58.418170 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:58.419009 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:58.419019 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:58.419585 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:58.420320 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:58.420339 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:58.421672 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:58.421679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:58.423559 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:58.423579 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:58.424821 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:58.424829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:58.425118 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:58.426151 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:58.426189 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:58.429561 (+ 3372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:58.433545 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:58.433559 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:58.433581 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:58.434320 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:58.434770 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:58.434777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:58.434784 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:58.435041 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:58.435649 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:58.435661 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:58.436191 (+ 530us) client_negotiation.cc:769] Sending connection context
0620 02:36:58.436520 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 02:36:58.437140 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":372,"threads_started":1}
I20260620 02:36:58.440658 10978 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34177
I20260620 02:36:58.441001 10978 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:58.441784 10978 heartbeater.cc:511] Master 127.0.97.62:34177 requested a full tablet report, sending...
I20260620 02:36:58.444059 10789 ts_manager.cc:194] Registered new tserver with Master: acc93d8f809e40d8ac421432067a11d1 (127.0.97.1:34073)
I20260620 02:36:58.445581 10789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59533
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:36:58.485234 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34177
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:58.841920 10984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:58.842320 10984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:58.842481 10984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:58.850157 10984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:58.850288 10984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:58.850476 10984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:36:58.861094 10984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:58.862530 10984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:58.864002 10984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:36:58.879153 10989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:58.879348 10990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:58.879317 10992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:58.880033 10984 server_base.cc:1034] running on GCE node
I20260620 02:36:58.880744 10984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:58.881757 10984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:58.883047 10984 hybrid_clock.cc:648] HybridClock initialized: now 1781923018882992 us; error 46 us; skew 500 ppm
I20260620 02:36:58.883445 10984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:58 dist-test-slave-rpg6 krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:58.887198 10984 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:36:58.889057 10984 webserver.cc:460] Webserver started at http://127.0.97.2:40863/ using document root <none> and password file <none>
I20260620 02:36:58.889935 10984 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:58.890100 10984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:58.890424 10984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:58.893962 10984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "abcdba2904a24fc593fa499d59611bec"
format_stamp: "Formatted at 2026-06-20 02:36:58 on dist-test-slave-rpg6"
server_key: "598e78dd5d30dd81a721667e9709cb48"
server_key_iv: "86db7e417b20a67a3ad4eceec1a20c13"
server_key_version: "encryptionkey@0"
I20260620 02:36:58.894848 10984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "abcdba2904a24fc593fa499d59611bec"
format_stamp: "Formatted at 2026-06-20 02:36:58 on dist-test-slave-rpg6"
server_key: "598e78dd5d30dd81a721667e9709cb48"
server_key_iv: "86db7e417b20a67a3ad4eceec1a20c13"
server_key_version: "encryptionkey@0"
I20260620 02:36:58.900826 10984 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:36:58.904692 10999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:58.906252 10984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:36:58.906464 10984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "abcdba2904a24fc593fa499d59611bec"
format_stamp: "Formatted at 2026-06-20 02:36:58 on dist-test-slave-rpg6"
server_key: "598e78dd5d30dd81a721667e9709cb48"
server_key_iv: "86db7e417b20a67a3ad4eceec1a20c13"
server_key_version: "encryptionkey@0"
I20260620 02:36:58.906711 10984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:58.926982 10984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:58.931692 10984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:58.932041 10984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:58.933151 10984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:58.934670 10984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:58.934788 10984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:58.934945 10984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:58.935020 10984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:58.971729 10984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35787
I20260620 02:36:58.971853 11111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35787 every 8 connection(s)
I20260620 02:36:58.973078 10984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:36:58.981658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 10984
I20260620 02:36:58.981869 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:36:58.982540 388 external_mini_cluster.cc:1428] Setting key 73a452f7771af7ab8d0b4c54bd23e162
Jun 20 02:36:58 dist-test-slave-rpg6 krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:36:58.997326 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:58.976144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51533 (local address 127.0.97.62:34177)
0620 02:36:58.976629 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:58.976633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:58.977712 (+ 1079us) server_negotiation.cc:407] Connection header received
0620 02:36:58.979823 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:58.980073 (+ 250us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:58.980220 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:58.980319 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:58.982103 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:58.983222 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:58.985495 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:58.986261 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:58.986416 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:58.990452 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:58.990462 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:58.990469 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:58.990518 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:58.993710 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:58.994749 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:58.994753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:58.994756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:58.994905 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:58.995442 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:58.995446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:58.995448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:58.995718 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:58.995895 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:58.996861 (+ 966us) server_negotiation.cc:299] Negotiation successful
0620 02:36:58.997043 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":286,"threads_started":1}
I20260620 02:36:58.998994 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:58.976605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34177 (local address 127.0.97.2:51533)
0620 02:36:58.977441 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:58.977559 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:58.979484 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:58.980607 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:58.980620 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:58.981236 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:58.981860 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:58.981885 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:58.983412 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:58.983419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:58.985320 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:58.985334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:58.986504 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:58.986510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:58.986765 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:58.987535 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:58.987564 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:58.990136 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:58.993892 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:58.993905 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:58.993927 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:58.994596 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:58.995028 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:58.995036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:58.995043 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:58.995301 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:58.995875 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:58.995887 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:58.996410 (+ 523us) client_negotiation.cc:769] Sending connection context
0620 02:36:58.996742 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 02:36:58.997342 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":344,"threads_started":1}
I20260620 02:36:59.000713 11112 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34177
I20260620 02:36:59.001199 11112 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:59.002025 11112 heartbeater.cc:511] Master 127.0.97.62:34177 requested a full tablet report, sending...
I20260620 02:36:59.003754 10789 ts_manager.cc:194] Registered new tserver with Master: abcdba2904a24fc593fa499d59611bec (127.0.97.2:35787)
I20260620 02:36:59.004726 10789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51533
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:36:59.048885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34177
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:36:59.411599 11118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:36:59.411957 11118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:36:59.412146 11118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:36:59.420154 11118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:36:59.420336 11118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:36:59.420511 11118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:36:59.432386 11118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:36:59.433841 11118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:36:59.435237 11118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 02:36:59.448827 10978 heartbeater.cc:503] Master 127.0.97.62:34177 was elected leader, sending a full tablet report...
W20260620 02:36:59.451318 11123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:59.451409 11124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:36:59.451761 11126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:36:59.452241 11118 server_base.cc:1034] running on GCE node
I20260620 02:36:59.452950 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:36:59.454136 11118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:36:59.455461 11118 hybrid_clock.cc:648] HybridClock initialized: now 1781923019455437 us; error 83 us; skew 500 ppm
I20260620 02:36:59.455945 11118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:36:59 dist-test-slave-rpg6 krb5kdc[10744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:36:59.459733 11118 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:36:59.461561 11118 webserver.cc:460] Webserver started at http://127.0.97.3:38671/ using document root <none> and password file <none>
I20260620 02:36:59.462340 11118 fs_manager.cc:359] Metadata directory not provided
I20260620 02:36:59.462455 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:36:59.462704 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:36:59.465898 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "fa54381f7a01416687b16bfaae03b3c7"
format_stamp: "Formatted at 2026-06-20 02:36:59 on dist-test-slave-rpg6"
server_key: "5a32886dfb5ba5489071278c0aa3c7f3"
server_key_iv: "2832e3491aea5579e2ea31eb1c56530a"
server_key_version: "encryptionkey@0"
I20260620 02:36:59.466712 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fa54381f7a01416687b16bfaae03b3c7"
format_stamp: "Formatted at 2026-06-20 02:36:59 on dist-test-slave-rpg6"
server_key: "5a32886dfb5ba5489071278c0aa3c7f3"
server_key_iv: "2832e3491aea5579e2ea31eb1c56530a"
server_key_version: "encryptionkey@0"
I20260620 02:36:59.473232 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 02:36:59.477773 11133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:59.479885 11118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:36:59.480123 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "fa54381f7a01416687b16bfaae03b3c7"
format_stamp: "Formatted at 2026-06-20 02:36:59 on dist-test-slave-rpg6"
server_key: "5a32886dfb5ba5489071278c0aa3c7f3"
server_key_iv: "2832e3491aea5579e2ea31eb1c56530a"
server_key_version: "encryptionkey@0"
I20260620 02:36:59.480358 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:36:59.500903 11118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:36:59.506965 11118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:36:59.507407 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:36:59.508510 11118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:36:59.510759 11118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:36:59.511085 11118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:59.511339 11118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:36:59.511399 11118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:36:59.551074 11118 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40947
I20260620 02:36:59.551211 11245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40947 every 8 connection(s)
I20260620 02:36:59.553048 11118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:36:59.555994 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 11118
I20260620 02:36:59.556224 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:36:59.556720 388 external_mini_cluster.cc:1428] Setting key 7018a247d1718f62ba5b0da62089edd9
Jun 20 02:36:59 dist-test-slave-rpg6 krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:36:59.580938 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.558055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32819 (local address 127.0.97.62:34177)
0620 02:36:59.560266 (+ 2211us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:59.560271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:59.560290 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:36:59.562258 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:59.562263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:59.562362 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:59.562475 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:59.564563 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.565468 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.567991 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.568483 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.568615 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.572786 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:59.572792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:59.572795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:59.572833 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:59.576469 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:59.577455 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:59.577460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:59.577462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:59.577589 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:59.578026 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:59.578030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:59.578032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:59.578378 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:59.578564 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:59.580136 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0620 02:36:59.580460 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2099,"thread_start_us":1981,"threads_started":1}
I20260620 02:36:59.581686 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.557135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34177 (local address 127.0.97.3:32819)
0620 02:36:59.558366 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:59.558446 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:59.561901 (+ 3455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:59.562884 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:59.562895 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:59.563531 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:36:59.564324 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.564346 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.566105 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.566114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.567659 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.567683 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.568769 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.568775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.569007 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.569814 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:36:59.569844 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:36:59.572479 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:36:59.576667 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:59.576676 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:59.576694 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:59.577284 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:59.577701 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:36:59.577706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:36:59.577709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:36:59.577902 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:36:59.578638 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:36:59.578649 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:36:59.579463 (+ 814us) client_negotiation.cc:769] Sending connection context
0620 02:36:59.579778 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 02:36:59.580403 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":431,"threads_started":1}
I20260620 02:36:59.583489 11246 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34177
I20260620 02:36:59.583940 11246 heartbeater.cc:464] Registering TS with master...
I20260620 02:36:59.584769 11246 heartbeater.cc:511] Master 127.0.97.62:34177 requested a full tablet report, sending...
I20260620 02:36:59.586596 10789 ts_manager.cc:194] Registered new tserver with Master: fa54381f7a01416687b16bfaae03b3c7 (127.0.97.3:40947)
I20260620 02:36:59.587985 10789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:32819
I20260620 02:36:59.589514 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:36:59.611452 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.595568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.0.97.62:34177)
0620 02:36:59.595945 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:59.595955 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:59.596791 (+ 836us) server_negotiation.cc:407] Connection header received
0620 02:36:59.597060 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:59.597064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:59.597140 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:59.597268 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:59.599261 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.600238 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.601828 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.602327 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.602449 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.605412 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:59.605417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:59.605420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:59.605462 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:59.608355 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:59.609172 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:59.609177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:59.609179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:59.609389 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:59.609769 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:59.609772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:59.609774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:59.610116 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:36:59.610369 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:59.610800 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 02:36:59.611100 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 02:36:59.616300 10789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:36:59.623909 10789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:36:59.650384 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.639477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34073 (local address 127.0.0.1:54066)
0620 02:36:59.641172 (+ 1695us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:59.641191 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:59.641306 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:59.643133 (+ 1827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:59.643141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:59.643164 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:59.643785 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.643800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.645618 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.645623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.647932 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.647951 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.649339 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.649346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.649565 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.649623 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 02:36:59.649767 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 02:36:59.649972 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1579,"thread_start_us":1422,"threads_started":1}
I20260620 02:36:59.650817 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.638854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35787 (local address 127.0.0.1:34544)
0620 02:36:59.640382 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:59.640420 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:59.640613 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:59.642679 (+ 2066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:59.642691 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:59.642728 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:59.643286 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.643298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.644977 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.644982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.647702 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.647719 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.649723 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.649732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.649907 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.649936 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 02:36:59.650006 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 02:36:59.650149 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1275,"mutex_wait_us":27,"thread_start_us":1126,"threads_started":1}
I20260620 02:36:59.651525 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.640292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.0.97.1:34073)
0620 02:36:59.642278 (+ 1986us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:59.642286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:59.642317 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:36:59.642416 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:59.642423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:59.642661 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:59.642880 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:59.644017 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.645372 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.648107 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.649115 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.649286 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.649549 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:59.650954 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0620 02:36:59.651164 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1795,"thread_start_us":1627,"threads_started":1}
I20260620 02:36:59.651880 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.639088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.0.97.2:35787)
0620 02:36:59.641887 (+ 2799us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:59.641896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:59.641927 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:36:59.642029 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:59.642040 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:59.642372 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:59.643311 (+ 939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:59.643531 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.644799 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.647924 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.649493 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.650986 (+ 1493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.651260 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:59.651332 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 02:36:59.651516 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2594,"thread_start_us":2427,"threads_started":1}
I20260620 02:36:59.653198 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.641954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40947 (local address 127.0.0.1:55144)
0620 02:36:59.643000 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:59.643022 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:59.643147 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:59.644754 (+ 1607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:59.644759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:59.644788 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:59.645282 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.645293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.647146 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.647152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.649284 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.649300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.651602 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.651609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.652557 (+ 948us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.652592 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:36:59.652700 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 02:36:59.652845 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":730,"threads_started":1}
I20260620 02:36:59.653524 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.641189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.0.97.3:40947)
0620 02:36:59.642530 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:59.642537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:59.643294 (+ 757us) server_negotiation.cc:407] Connection header received
0620 02:36:59.643399 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:59.643407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:59.643635 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:59.644726 (+ 1091us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:59.645492 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.646947 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.649496 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.651064 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.651242 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.651452 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:59.652818 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0620 02:36:59.652998 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1189,"thread_start_us":1037,"threads_started":1}
I20260620 02:36:59.655733 11047 tablet_service.cc:1467] Processing CreateTablet for tablet 463c5561b6ee45b4b779dd37a24faac9 (DEFAULT_TABLE table=test-table [id=43d94b748a4249f885dcfbdd3d6896b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:59.655767 10913 tablet_service.cc:1467] Processing CreateTablet for tablet 463c5561b6ee45b4b779dd37a24faac9 (DEFAULT_TABLE table=test-table [id=43d94b748a4249f885dcfbdd3d6896b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:59.657126 11047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463c5561b6ee45b4b779dd37a24faac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:59.657240 10913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463c5561b6ee45b4b779dd37a24faac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:59.658352 11181 tablet_service.cc:1467] Processing CreateTablet for tablet 463c5561b6ee45b4b779dd37a24faac9 (DEFAULT_TABLE table=test-table [id=43d94b748a4249f885dcfbdd3d6896b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:36:59.659958 11181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463c5561b6ee45b4b779dd37a24faac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:36:59.669147 11265 tablet_bootstrap.cc:492] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: Bootstrap starting.
I20260620 02:36:59.673007 11266 tablet_bootstrap.cc:492] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1: Bootstrap starting.
I20260620 02:36:59.673604 11265 tablet_bootstrap.cc:654] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:59.675127 11265 log.cc:824] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:59.675909 11267 tablet_bootstrap.cc:492] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: Bootstrap starting.
I20260620 02:36:59.677115 11266 tablet_bootstrap.cc:654] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:59.678638 11266 log.cc:824] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:59.679952 11267 tablet_bootstrap.cc:654] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: Neither blocks nor log segments found. Creating new log.
I20260620 02:36:59.680050 11265 tablet_bootstrap.cc:492] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: No bootstrap required, opened a new log
I20260620 02:36:59.680434 11265 ts_tablet_manager.cc:1397] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260620 02:36:59.681329 11267 log.cc:824] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: Log is configured to *not* fsync() on all Append() calls
I20260620 02:36:59.682533 11266 tablet_bootstrap.cc:492] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1: No bootstrap required, opened a new log
I20260620 02:36:59.682847 11266 ts_tablet_manager.cc:1397] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 02:36:59.684784 11267 tablet_bootstrap.cc:492] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: No bootstrap required, opened a new log
I20260620 02:36:59.685084 11267 ts_tablet_manager.cc:1397] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 02:36:59.690871 11266 raft_consensus.cc:348] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcdba2904a24fc593fa499d59611bec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35787 } } peers { permanent_uuid: "acc93d8f809e40d8ac421432067a11d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34073 } } peers { permanent_uuid: "fa54381f7a01416687b16bfaae03b3c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40947 } }
I20260620 02:36:59.691296 11266 raft_consensus.cc:374] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:59.691387 11266 raft_consensus.cc:729] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc93d8f809e40d8ac421432067a11d1, State: Initialized, Role: FOLLOWER
I20260620 02:36:59.691622 11265 raft_consensus.cc:348] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcdba2904a24fc593fa499d59611bec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35787 } } peers { permanent_uuid: "acc93d8f809e40d8ac421432067a11d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34073 } } peers { permanent_uuid: "fa54381f7a01416687b16bfaae03b3c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40947 } }
I20260620 02:36:59.692086 11265 raft_consensus.cc:374] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:59.692195 11265 raft_consensus.cc:729] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abcdba2904a24fc593fa499d59611bec, State: Initialized, Role: FOLLOWER
I20260620 02:36:59.692224 11266 consensus_queue.cc:260] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcdba2904a24fc593fa499d59611bec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35787 } } peers { permanent_uuid: "acc93d8f809e40d8ac421432067a11d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34073 } } peers { permanent_uuid: "fa54381f7a01416687b16bfaae03b3c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40947 } }
I20260620 02:36:59.692790 11265 consensus_queue.cc:260] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcdba2904a24fc593fa499d59611bec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35787 } } peers { permanent_uuid: "acc93d8f809e40d8ac421432067a11d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34073 } } peers { permanent_uuid: "fa54381f7a01416687b16bfaae03b3c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40947 } }
I20260620 02:36:59.694555 11266 ts_tablet_manager.cc:1428] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 02:36:59.695642 11112 heartbeater.cc:503] Master 127.0.97.62:34177 was elected leader, sending a full tablet report...
I20260620 02:36:59.696331 11265 ts_tablet_manager.cc:1428] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260620 02:36:59.696930 11267 raft_consensus.cc:348] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcdba2904a24fc593fa499d59611bec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35787 } } peers { permanent_uuid: "acc93d8f809e40d8ac421432067a11d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34073 } } peers { permanent_uuid: "fa54381f7a01416687b16bfaae03b3c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40947 } }
I20260620 02:36:59.697455 11267 raft_consensus.cc:374] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:36:59.697549 11267 raft_consensus.cc:729] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa54381f7a01416687b16bfaae03b3c7, State: Initialized, Role: FOLLOWER
I20260620 02:36:59.698238 11267 consensus_queue.cc:260] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcdba2904a24fc593fa499d59611bec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35787 } } peers { permanent_uuid: "acc93d8f809e40d8ac421432067a11d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34073 } } peers { permanent_uuid: "fa54381f7a01416687b16bfaae03b3c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40947 } }
I20260620 02:36:59.700001 11246 heartbeater.cc:503] Master 127.0.97.62:34177 was elected leader, sending a full tablet report...
I20260620 02:36:59.700817 11267 ts_tablet_manager.cc:1428] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260620 02:36:59.725485 11113 tablet.cc:2369] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:36:59.805299 11247 tablet.cc:2369] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:36:59.832321 11272 raft_consensus.cc:482] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:36:59.832640 11272 raft_consensus.cc:504] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcdba2904a24fc593fa499d59611bec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35787 } } peers { permanent_uuid: "acc93d8f809e40d8ac421432067a11d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34073 } } peers { permanent_uuid: "fa54381f7a01416687b16bfaae03b3c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40947 } }
I20260620 02:36:59.834579 11272 leader_election.cc:288] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acc93d8f809e40d8ac421432067a11d1 (127.0.97.1:34073), fa54381f7a01416687b16bfaae03b3c7 (127.0.97.3:40947)
I20260620 02:36:59.842979 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.835580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58613 (local address 127.0.97.1:34073)
0620 02:36:59.835900 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:59.835909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:59.836633 (+ 724us) server_negotiation.cc:407] Connection header received
0620 02:36:59.836895 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:59.836900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:59.836966 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:59.837076 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:59.837806 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.838586 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.840859 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.841952 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.842109 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.842236 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:59.842598 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 02:36:59.842747 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 02:36:59.842971 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.835104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37715 (local address 127.0.97.3:40947)
0620 02:36:59.835359 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:59.835367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:59.836784 (+ 1417us) server_negotiation.cc:407] Connection header received
0620 02:36:59.837000 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:59.837005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:59.837064 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:59.837160 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:36:59.837881 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.838640 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.840813 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.841784 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.841931 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.842038 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:59.842532 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 02:36:59.842677 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 02:36:59.843137 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.835488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40947 (local address 127.0.97.2:37715)
0620 02:36:59.836661 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:59.836676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:59.836841 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:59.837183 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:59.837187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:59.837216 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:59.837673 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.837682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.838788 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.838794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.840616 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.840634 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.842073 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.842081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.842336 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.842364 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 02:36:59.842495 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 02:36:59.842653 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":73,"thread_start_us":824,"threads_started":1}
I20260620 02:36:59.844005 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.835263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34073 (local address 127.0.97.2:58613)
0620 02:36:59.836473 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0620 02:36:59.836497 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:36:59.836724 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:36:59.837110 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:36:59.837115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:36:59.837142 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:36:59.837630 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.837642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.838746 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.838754 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.840653 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:36:59.840667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.842224 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.842230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:36:59.842398 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.842431 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:36:59.843217 (+ 786us) client_negotiation.cc:240] Negotiation successful
0620 02:36:59.843581 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":27,"thread_start_us":493,"threads_started":1}
I20260620 02:36:59.844585 11201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463c5561b6ee45b4b779dd37a24faac9" candidate_uuid: "abcdba2904a24fc593fa499d59611bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa54381f7a01416687b16bfaae03b3c7" is_pre_election: true
I20260620 02:36:59.845283 11201 raft_consensus.cc:2454] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abcdba2904a24fc593fa499d59611bec in term 0.
I20260620 02:36:59.845124 10933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463c5561b6ee45b4b779dd37a24faac9" candidate_uuid: "abcdba2904a24fc593fa499d59611bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc93d8f809e40d8ac421432067a11d1" is_pre_election: true
I20260620 02:36:59.845718 10933 raft_consensus.cc:2454] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abcdba2904a24fc593fa499d59611bec in term 0.
I20260620 02:36:59.846081 11001 leader_election.cc:302] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abcdba2904a24fc593fa499d59611bec, fa54381f7a01416687b16bfaae03b3c7; no voters:
I20260620 02:36:59.846551 11272 raft_consensus.cc:2781] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:36:59.846695 11272 raft_consensus.cc:482] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:36:59.846829 11272 raft_consensus.cc:3037] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:59.849107 11272 raft_consensus.cc:504] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcdba2904a24fc593fa499d59611bec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35787 } } peers { permanent_uuid: "acc93d8f809e40d8ac421432067a11d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34073 } } peers { permanent_uuid: "fa54381f7a01416687b16bfaae03b3c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40947 } }
I20260620 02:36:59.849761 11272 leader_election.cc:288] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [CANDIDATE]: Term 1 election: Requested vote from peers acc93d8f809e40d8ac421432067a11d1 (127.0.97.1:34073), fa54381f7a01416687b16bfaae03b3c7 (127.0.97.3:40947)
I20260620 02:36:59.850404 11201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463c5561b6ee45b4b779dd37a24faac9" candidate_uuid: "abcdba2904a24fc593fa499d59611bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa54381f7a01416687b16bfaae03b3c7"
I20260620 02:36:59.850520 10933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463c5561b6ee45b4b779dd37a24faac9" candidate_uuid: "abcdba2904a24fc593fa499d59611bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc93d8f809e40d8ac421432067a11d1"
I20260620 02:36:59.850756 10933 raft_consensus.cc:3037] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:59.850687 11201 raft_consensus.cc:3037] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:36:59.853984 11201 raft_consensus.cc:2454] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abcdba2904a24fc593fa499d59611bec in term 1.
I20260620 02:36:59.854041 10933 raft_consensus.cc:2454] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abcdba2904a24fc593fa499d59611bec in term 1.
I20260620 02:36:59.854774 11001 leader_election.cc:302] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abcdba2904a24fc593fa499d59611bec, fa54381f7a01416687b16bfaae03b3c7; no voters:
I20260620 02:36:59.855185 11272 raft_consensus.cc:2781] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:36:59.855736 11272 raft_consensus.cc:686] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 1 LEADER]: Becoming Leader. State: Replica: abcdba2904a24fc593fa499d59611bec, State: Running, Role: LEADER
I20260620 02:36:59.856590 11272 consensus_queue.cc:237] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcdba2904a24fc593fa499d59611bec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35787 } } peers { permanent_uuid: "acc93d8f809e40d8ac421432067a11d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34073 } } peers { permanent_uuid: "fa54381f7a01416687b16bfaae03b3c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40947 } }
I20260620 02:36:59.862253 10788 catalog_manager.cc:5495] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec reported cstate change: term changed from 0 to 1, leader changed from <none> to abcdba2904a24fc593fa499d59611bec (127.0.97.2). New cstate: current_term: 1 leader_uuid: "abcdba2904a24fc593fa499d59611bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcdba2904a24fc593fa499d59611bec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc93d8f809e40d8ac421432067a11d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa54381f7a01416687b16bfaae03b3c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40947 } health_report { overall_health: UNKNOWN } } }
I20260620 02:36:59.904711 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.898336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.0.97.2:35787)
0620 02:36:59.898608 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:59.898616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:59.898747 (+ 131us) server_negotiation.cc:407] Connection header received
0620 02:36:59.899010 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:59.899014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:59.899106 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:59.899607 (+ 501us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:36:59.900264 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.901037 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.902391 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.902804 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.902947 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.903433 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:36:59.904096 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:36:59.904260 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:36:59.904338 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 02:36:59.904424 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 02:36:59.913868 11201 raft_consensus.cc:1261] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7 [term 1 FOLLOWER]: Refusing update from remote peer abcdba2904a24fc593fa499d59611bec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:36:59.914140 10933 raft_consensus.cc:1261] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1 [term 1 FOLLOWER]: Refusing update from remote peer abcdba2904a24fc593fa499d59611bec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 02:36:59.914325 10979 tablet.cc:2369] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:36:59.915025 11276 consensus_queue.cc:1035] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa54381f7a01416687b16bfaae03b3c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:59.915705 11272 consensus_queue.cc:1035] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc93d8f809e40d8ac421432067a11d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:36:59.944763 10789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:59.945046 10789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:36:59.948743 10789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca2274b637aa44439a31a2900af8a093: Sending DeleteTablet for 3 replicas of tablet 463c5561b6ee45b4b779dd37a24faac9
I20260620 02:36:59.950177 11181 tablet_service.cc:1514] Processing DeleteTablet for tablet 463c5561b6ee45b4b779dd37a24faac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55144
I20260620 02:36:59.950167 10913 tablet_service.cc:1514] Processing DeleteTablet for tablet 463c5561b6ee45b4b779dd37a24faac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54066
I20260620 02:36:59.951390 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 10850
I20260620 02:36:59.951508 11047 tablet_service.cc:1514] Processing DeleteTablet for tablet 463c5561b6ee45b4b779dd37a24faac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:36:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34544
I20260620 02:36:59.977787 11291 tablet_replica.cc:330] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: stopping tablet replica
I20260620 02:36:59.978772 11291 raft_consensus.cc:2229] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:36:59.979586 11289 tablet_replica.cc:330] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: stopping tablet replica
I20260620 02:36:59.980937 11291 raft_consensus.cc:2258] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:59.981326 11289 raft_consensus.cc:2229] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:36:59.982015 11289 raft_consensus.cc:2258] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:36:59.986090 11291 ts_tablet_manager.cc:1905] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:36:59 dist-test-slave-rpg6 krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:36:59.988147 11289 ts_tablet_manager.cc:1905] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:36:59.998188 11291 ts_tablet_manager.cc:1918] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:36:59.998421 11291 log.cc:1196] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/463c5561b6ee45b4b779dd37a24faac9
I20260620 02:36:59.998678 11290 tablet_replica.cc:330] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1: stopping tablet replica
I20260620 02:36:59.999035 11291 ts_tablet_manager.cc:1939] T 463c5561b6ee45b4b779dd37a24faac9 P abcdba2904a24fc593fa499d59611bec: Deleting consensus metadata
I20260620 02:36:59.999378 11290 raft_consensus.cc:2229] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:37:00.000010 11290 raft_consensus.cc:2258] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:37:00.001053 10774 catalog_manager.cc:4841] TS abcdba2904a24fc593fa499d59611bec (127.0.97.2:35787): tablet 463c5561b6ee45b4b779dd37a24faac9 (table test-table [id=43d94b748a4249f885dcfbdd3d6896b1]) successfully deleted
I20260620 02:37:00.003293 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:36:59.952905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.0.97.1:34073)
0620 02:36:59.953083 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 02:36:59.953089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:36:59.962997 (+ 9908us) server_negotiation.cc:407] Connection header received
0620 02:36:59.963506 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:36:59.963514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:36:59.963619 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:36:59.963708 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:36:59.965519 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.966746 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.968661 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:36:59.969354 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:36:59.969460 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:36:59.993038 (+ 23578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:36:59.993059 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:36:59.993081 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:36:59.993152 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:36:59.997368 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:59.998470 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:59.998481 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:59.998484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:59.998675 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:36:59.999199 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:36:59.999204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:36:59.999207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:36:59.999682 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:37:00.000062 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:00.002285 (+ 2223us) server_negotiation.cc:299] Negotiation successful
0620 02:37:00.002766 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:37:00.003750 11290 ts_tablet_manager.cc:1905] T 463c5561b6ee45b4b779dd37a24faac9 P acc93d8f809e40d8ac421432067a11d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:37:00.004432 10973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:37:00.006081 11289 ts_tablet_manager.cc:1918] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:37:00.006265 11289 log.cc:1196] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/463c5561b6ee45b4b779dd37a24faac9
I20260620 02:37:00.006778 11289 ts_tablet_manager.cc:1939] T 463c5561b6ee45b4b779dd37a24faac9 P fa54381f7a01416687b16bfaae03b3c7: Deleting consensus metadata
I20260620 02:37:00.008730 10774 catalog_manager.cc:4841] TS fa54381f7a01416687b16bfaae03b3c7 (127.0.97.3:40947): tablet 463c5561b6ee45b4b779dd37a24faac9 (table test-table [id=43d94b748a4249f885dcfbdd3d6896b1]) successfully deleted
I20260620 02:37:00.105633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 10850
W20260620 02:37:00.131649 10776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34073 (error 108)
I20260620 02:37:00.132436 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 10984
I20260620 02:37:00.133675 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:00.133172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34073 (local address 127.0.0.1:54086)
0620 02:37:00.133381 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:00.133503 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 02:37:00.133925 10776 catalog_manager.cc:4584] TS acc93d8f809e40d8ac421432067a11d1 (127.0.97.1:34073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 463c5561b6ee45b4b779dd37a24faac9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34073: connect: Connection refused (error 111)
Jun 20 02:37:00 dist-test-slave-rpg6 krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:37:00.152053 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:00.133482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.0.97.2:35787)
0620 02:37:00.133662 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:00.133670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:00.133693 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:37:00.133758 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:00.133789 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:00.133872 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:00.133996 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:37:00.135656 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:00.136601 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:00.138742 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:00.139285 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:00.139414 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:00.145364 (+ 5950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:37:00.145382 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:37:00.145399 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:37:00.145448 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:37:00.148946 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:00.149818 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:00.149826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:00.149829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:00.149983 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:00.150365 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:00.150372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:00.150375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:00.150770 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:37:00.151134 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:00.151371 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 02:37:00.151622 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:37:00.152817 11107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:37:00.187565 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:00.187089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34073 (local address 127.0.0.1:54098)
0620 02:37:00.187306 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:00.187418 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 02:37:00.188771 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:00.188387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34073 (local address 127.0.0.1:54104)
0620 02:37:00.188536 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:00.188606 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 02:37:00.249908 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:00.249063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34073 (local address 127.0.0.1:54108)
0620 02:37:00.249429 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:00.249576 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 02:37:00.251514 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:00.251005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34073 (local address 127.0.0.1:54124)
0620 02:37:00.251266 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:00.251360 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 02:37:00.273512 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 10984
I20260620 02:37:00.294431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 11118
Jun 20 02:37:00 dist-test-slave-rpg6 krb5kdc[10744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:37:00.312409 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:00.295342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.0.97.3:40947)
0620 02:37:00.295550 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:00.295557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:00.295967 (+ 410us) server_negotiation.cc:407] Connection header received
0620 02:37:00.296177 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:00.296184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:00.296268 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:00.296379 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:37:00.297975 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:00.298866 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:00.300397 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:00.300796 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:00.300911 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:00.305744 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:37:00.305762 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:37:00.305779 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:37:00.305835 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:37:00.309237 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:00.310067 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:00.310076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:00.310078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:00.310315 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:00.310676 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:00.310681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:00.310684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:00.311055 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:37:00.311361 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:00.311671 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 02:37:00.311924 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 02:37:00.313139 11241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:37:00.332442 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:00.331799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34073 (local address 127.0.0.1:54126)
0620 02:37:00.332106 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:00.332216 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260620 02:37:00.333726 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:00.333240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34073 (local address 127.0.0.1:54134)
0620 02:37:00.333466 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:00.333571 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 02:37:00.406949 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 11118
I20260620 02:37:00.424221 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 10757
I20260620 02:37:00.425556 10819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:37:00.536659 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 10757
2026-06-20T02:37:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4701 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:37:00 dist-test-slave-rpg6 krb5kdc[11300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:37:00 dist-test-slave-rpg6 krb5kdc[11300](info): set up 2 sockets
Jun 20 02:37:00 dist-test-slave-rpg6 krb5kdc[11300](info): commencing operation
krb5kdc: starting...
W20260620 02:37:01.660867 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:37:01 dist-test-slave-rpg6 krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:37:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:37:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 02:37:02.036327 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRSL2lK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/postgres -l logfile start
2026-06-20 02:37:05.144 UTC [11330] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 02:37:05.145 UTC [11330] LOG: listening on IPv4 address "127.0.97.20", port 46235
2026-06-20 02:37:05.149 UTC [11330] LOG: listening on Unix socket "/tmp/.s.PGSQL.46235"
2026-06-20 02:37:05.175 UTC [11333] LOG: database system was shut down at 2026-06-20 02:37:03 UTC
2026-06-20 02:37:05.180 UTC [11330] LOG: database system is ready to accept connections
I20260620 02:37:06.184792 388 mini_postgres.cc:96] Postgres bound to 46235
2026-06-20 02:37:06.204 UTC [11341] FATAL: database "slave" does not exist
127.0.97.20:46235 - accepting connections
I20260620 02:37:06.205979 388 mini_ranger.cc:161] Starting Ranger...
I20260620 02:37:06.238556 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 02:37:06.933506 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 02:37:06.933698 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-admin
2026-06-20 02:37:07,263 [I] DB FLAVOR :POSTGRES
2026-06-20 02:37:07,264 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:37:07,264 [I] Checking connection..
2026-06-20 02:37:07,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 02:37:07,556 [I] Checking connection passed.
2026-06-20 02:37:07,556 [I] --------- Verifying version history table ---------
2026-06-20 02:37:07,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:37:07,813 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:37:07,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:37:08,087 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:37:08,087 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 02:37:08,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 02:37:08.352 UTC [11405] WARNING: there is no transaction in progress
2026-06-20 02:37:08,360 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 02:37:08,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:37:08,643 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 02:37:08,643 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:37:08,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 02:37:08,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 02:37:09,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rpg6.c.gcp-upstream.internal','N') ;"
2026-06-20 02:37:09,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:37:09,728 [I] Table x_portal_user does not exist in database ranger
2026-06-20 02:37:09,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:37:10,008 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 02:37:10,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:37:10,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:37:10,544 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 02:37:10,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 02:37:11.680 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.701 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.720 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.740 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.763 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.784 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.852 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.861 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.879 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.891 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.903 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.921 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.933 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.941 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.950 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:11.959 UTC [11513] WARNING: there is no transaction in progress
2026-06-20 02:37:12,853 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 02:37:12,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 02:37:13,104 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 02:37:13,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:37:13,395 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 02:37:13,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:37:13,662 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 02:37:13,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:37:13,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:37:14,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rpg6.c.gcp-upstream.internal';"
2026-06-20 02:37:14,481 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 02:37:14,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 02:37:14,747 [I] DB_PATCHES have already been applied
I20260620 02:37:14.753650 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRSL2lK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 02:37:14.753820 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 02:37:14.770794 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:37467
Jun 20, 2026 2:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 02:37:15.955018 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.749s user 0.000s sys 0.051s
Jun 20, 2026 2:37:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:37:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:37:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:37:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:37:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 02:37:39.266664 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 02:37:39 dist-test-slave-rpg6 krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923059, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 02:37:39.462277 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 02:37:39.485399 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 02:37:40.589180 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 02:37:40.589337 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-kms
I20260620 02:37:40.589490 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:46235
2026-06-20 02:37:41,053 [I] DB FLAVOR :POSTGRES
2026-06-20 02:37:41,053 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:37:41,053 [I] Checking connection
2026-06-20 02:37:41,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 02:37:41,338 [I] connection success
2026-06-20 02:37:41,338 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 02:37:41,338 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 02:37:41,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 02:37:41,612 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 02:37:41,612 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:37:41,613 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 02:37:41,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46235/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 02:37:41,920 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 02:37:42.480029 388 mini_ranger_kms.cc:310] Created kms service
I20260620 02:37:42.680173 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 02:37:42.727165 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 02:37:42 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923059, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:37:43.148026 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 02:37:43.148339 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 02:37:43.148397 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 02:37:43.169968 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11694
I20260620 02:37:43.170190 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:55391
Jun 20, 2026 2:37:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:37:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:37:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:37:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:37:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:37:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 02:37:45 dist-test-slave-rpg6 krb5kdc[11300](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781923065, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:37:45.618789 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.157s user 0.126s sys 0.357s
I20260620 02:37:45.619108 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 02:37:45.619335 388 mini_ranger_kms.cc:401] 127.0.97.20:55391/kms/v1/keys
Jun 20 02:37:46 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923065, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:37:46 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923065, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:37:49 dist-test-slave-rpg6 krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:37:49.196775 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43537
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43537
--ranger_config_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55391/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:37:49.572762 11744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:37:49.573102 11744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:37:49.573256 11744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:37:49.580911 11744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:37:49.581043 11744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:37:49.581095 11744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:37:49.581185 11744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:37:49.581254 11744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:37:49.592286 11744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:55391/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43537
--ranger_config_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:37:49.593758 11744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:37:49.595146 11744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:37:49.608419 11749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:37:49.608402 11750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:37:49.608397 11752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:37:49.609431 11744 server_base.cc:1034] running on GCE node
I20260620 02:37:49.610302 11744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:37:49.612404 11744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:37:49.613713 11744 hybrid_clock.cc:648] HybridClock initialized: now 1781923069613627 us; error 73 us; skew 500 ppm
I20260620 02:37:49.614234 11744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:37:49 dist-test-slave-rpg6 krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:37:49.617733 11744 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:37:49.619473 11744 webserver.cc:460] Webserver started at http://127.0.97.62:40279/ using document root <none> and password file <none>
I20260620 02:37:49.620231 11744 fs_manager.cc:359] Metadata directory not provided
I20260620 02:37:49.620337 11744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:37:49.620589 11744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:37:49 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:37:49.692029 11744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "869ed600e3924b4a8df3c8af9bbf96ee"
format_stamp: "Formatted at 2026-06-20 02:37:49 on dist-test-slave-rpg6"
server_key: "28f8e1e383ac3555b1ba7b852c5f7749"
server_key_iv: "54db72d4076e99f1546fa92fec8ed919"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:49.692986 11744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "869ed600e3924b4a8df3c8af9bbf96ee"
format_stamp: "Formatted at 2026-06-20 02:37:49 on dist-test-slave-rpg6"
server_key: "28f8e1e383ac3555b1ba7b852c5f7749"
server_key_iv: "54db72d4076e99f1546fa92fec8ed919"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:49.699003 11744 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:37:49.722007 11759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:37:49.723937 11744 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260620 02:37:49.724164 11744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "869ed600e3924b4a8df3c8af9bbf96ee"
format_stamp: "Formatted at 2026-06-20 02:37:49 on dist-test-slave-rpg6"
server_key: "28f8e1e383ac3555b1ba7b852c5f7749"
server_key_iv: "54db72d4076e99f1546fa92fec8ed919"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:49.724470 11744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:37:49.755338 11744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:37:49.760448 11744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:37:49.760798 11744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:37:49.784340 11744 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43537
I20260620 02:37:49.784399 11810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43537 every 8 connection(s)
I20260620 02:37:49.786067 11744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:37:49.789021 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 11744
I20260620 02:37:49.789263 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:37:49.790073 388 external_mini_cluster.cc:1428] Setting key 02d2cbc9a9861f7f9b9051af06755d63
I20260620 02:37:49.791008 11811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:37:49.807152 11811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee: Bootstrap starting.
Jun 20 02:37:49 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:37:49.812127 11811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee: Neither blocks nor log segments found. Creating new log.
I20260620 02:37:49.813683 11811 log.cc:824] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee: Log is configured to *not* fsync() on all Append() calls
I20260620 02:37:49.817767 11811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee: No bootstrap required, opened a new log
I20260620 02:37:49.819731 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:49.792568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.0.97.62:43537)
0620 02:37:49.793927 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:49.793943 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:49.794004 (+ 61us) server_negotiation.cc:407] Connection header received
0620 02:37:49.796803 (+ 2799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:49.796844 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:49.797519 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:49.798101 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:37:49.799499 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:49.801631 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:49.803700 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:49.804958 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:49.805193 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:49.809872 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:37:49.809884 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:37:49.809904 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:37:49.809954 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:37:49.814502 (+ 4548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:49.815554 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:49.815564 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:49.815575 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:49.815787 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:49.816227 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:49.816232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:49.816235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:49.816975 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:37:49.817333 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:49.817761 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 02:37:49.818216 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":833,"threads_started":1}
I20260620 02:37:49.827198 11811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed600e3924b4a8df3c8af9bbf96ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } }
I20260620 02:37:49.828233 11811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:37:49.828416 11811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869ed600e3924b4a8df3c8af9bbf96ee, State: Initialized, Role: FOLLOWER
I20260620 02:37:49.829083 11811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed600e3924b4a8df3c8af9bbf96ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } }
I20260620 02:37:49.829298 11811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:37:49.829463 11811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:37:49.829658 11811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:37:49.832008 11811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed600e3924b4a8df3c8af9bbf96ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } }
I20260620 02:37:49.832594 11811 leader_election.cc:302] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 869ed600e3924b4a8df3c8af9bbf96ee; no voters:
I20260620 02:37:49.833328 11811 leader_election.cc:288] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:37:49.833483 11816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:37:49.834237 11816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [term 1 LEADER]: Becoming Leader. State: Replica: 869ed600e3924b4a8df3c8af9bbf96ee, State: Running, Role: LEADER
I20260620 02:37:49.834828 11816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed600e3924b4a8df3c8af9bbf96ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } }
I20260620 02:37:49.835693 11811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:37:49.838114 11818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869ed600e3924b4a8df3c8af9bbf96ee. Latest consensus state: current_term: 1 leader_uuid: "869ed600e3924b4a8df3c8af9bbf96ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed600e3924b4a8df3c8af9bbf96ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } }
I20260620 02:37:49.838649 11818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [sys.catalog]: This master's current role is: LEADER
I20260620 02:37:49.838605 11817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "869ed600e3924b4a8df3c8af9bbf96ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed600e3924b4a8df3c8af9bbf96ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43537 } } }
I20260620 02:37:49.839890 11817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee [sys.catalog]: This master's current role is: LEADER
I20260620 02:37:49.841446 11825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:37:49.847909 11825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:37:49.848834 11811 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 02:37:49.874485 11825 catalog_manager.cc:1347] Generated new cluster ID: 5f606745d0554feb95cd3ec341c387b1
I20260620 02:37:49.874630 11825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:37:49.893069 11825 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:37:49.895687 11825 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:37:49.913935 11825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee: Generated new TSK 0
I20260620 02:37:49.915196 11825 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:37:50.734074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55391/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43537
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:37:51 dist-test-slave-rpg6 krb5kdc[11300](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781923071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 02:37:51.167753 11856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:37:51.168201 11856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:37:51.168399 11856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:37:51.177527 11856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:37:51.177711 11856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:37:51.177894 11856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:37:51.190120 11856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:55391/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:37:51.191568 11856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:37:51.193228 11856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:37:51.211439 11867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:37:51.218976 11869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:37:51.219043 11866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:37:51.219769 11856 server_base.cc:1034] running on GCE node
I20260620 02:37:51.220701 11856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:37:51.222131 11856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:37:51.223484 11856 hybrid_clock.cc:648] HybridClock initialized: now 1781923071223400 us; error 81 us; skew 500 ppm
I20260620 02:37:51.223966 11856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:37:51 dist-test-slave-rpg6 krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:37:51.228899 11856 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:37:51.232316 11856 webserver.cc:460] Webserver started at http://127.0.97.1:40329/ using document root <none> and password file <none>
I20260620 02:37:51.233386 11856 fs_manager.cc:359] Metadata directory not provided
I20260620 02:37:51.233547 11856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:37:51.233981 11856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:37:51 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:37:51.262984 11856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "e7713a95b9544e55a17be8cb3027f9af"
format_stamp: "Formatted at 2026-06-20 02:37:51 on dist-test-slave-rpg6"
server_key: "69f4a9aa03f42228cb2af8e05e07b298"
server_key_iv: "7d11a085d9d24d712d0dea0e43c19d85"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:51.265010 11856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e7713a95b9544e55a17be8cb3027f9af"
format_stamp: "Formatted at 2026-06-20 02:37:51 on dist-test-slave-rpg6"
server_key: "69f4a9aa03f42228cb2af8e05e07b298"
server_key_iv: "7d11a085d9d24d712d0dea0e43c19d85"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:51.286405 11856 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.016s sys 0.004s
I20260620 02:37:51.316892 11876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:37:51.319406 11856 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.008s sys 0.000s
I20260620 02:37:51.319734 11856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "e7713a95b9544e55a17be8cb3027f9af"
format_stamp: "Formatted at 2026-06-20 02:37:51 on dist-test-slave-rpg6"
server_key: "69f4a9aa03f42228cb2af8e05e07b298"
server_key_iv: "7d11a085d9d24d712d0dea0e43c19d85"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:51.320182 11856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:37:51.370550 11856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:37:51.376989 11856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:37:51.377404 11856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:37:51.378803 11856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:37:51.385012 11856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:37:51.385167 11856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:37:51.385388 11856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:37:51.385476 11856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:37:51 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:37:51.943265 11856 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33213
I20260620 02:37:51.943337 11988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33213 every 8 connection(s)
I20260620 02:37:51.945251 11856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:37:51.955312 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 11856
I20260620 02:37:51.955540 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:37:51.956405 388 external_mini_cluster.cc:1428] Setting key 43de838029de0802e100d2ca742d98b2
Jun 20 02:37:51 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:37:51.986613 11991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:51.948955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39579 (local address 127.0.97.62:43537)
0620 02:37:51.962196 (+ 13241us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:51.962204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:51.962229 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:37:51.964979 (+ 2750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:51.964988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:51.965093 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:51.965180 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:37:51.967942 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:51.969177 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:51.972051 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:51.972821 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:51.972926 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:51.977923 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:37:51.977934 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:37:51.977939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:37:51.977990 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:37:51.981812 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:51.983042 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:51.983049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:51.983053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:51.983226 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:51.983856 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:51.983861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:51.983864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:51.984240 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:37:51.984379 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:51.985440 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0620 02:37:51.985698 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13103,"thread_start_us":12923,"threads_started":1}
I20260620 02:37:51.989019 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:51.949521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43537 (local address 127.0.97.1:39579)
0620 02:37:51.961293 (+ 11772us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:51.961378 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 02:37:51.964517 (+ 3139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:37:51.965891 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:37:51.965908 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:37:51.966640 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:37:51.967626 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:51.967660 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:51.969477 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:51.969486 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:51.971818 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:51.971871 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:51.973315 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:51.973323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:51.973676 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:51.974560 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:37:51.974593 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:37:51.977547 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:37:51.982088 (+ 4541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:37:51.982100 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:37:51.982122 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:37:51.982866 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:37:51.983393 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:37:51.983398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:37:51.983402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:37:51.983684 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:37:51.984480 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:37:51.984490 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:37:51.984973 (+ 483us) client_negotiation.cc:769] Sending connection context
0620 02:37:51.986906 (+ 1933us) client_negotiation.cc:240] Negotiation successful
0620 02:37:51.987515 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11274,"thread_start_us":10975,"threads_started":1}
I20260620 02:37:51.991101 11989 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43537
I20260620 02:37:51.991448 11989 heartbeater.cc:464] Registering TS with master...
I20260620 02:37:51.992336 11989 heartbeater.cc:511] Master 127.0.97.62:43537 requested a full tablet report, sending...
I20260620 02:37:51.995039 11776 ts_manager.cc:194] Registered new tserver with Master: e7713a95b9544e55a17be8cb3027f9af (127.0.97.1:33213)
I20260620 02:37:51.997434 11776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39579
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:37:52.040868 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55391/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43537
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:37:52 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 02:37:52.578747 11995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:37:52.579304 11995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:37:52.579514 11995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:37:52.591883 11995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:37:52.592156 11995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:37:52.592413 11995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:37:52.609918 11995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:55391/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:37:52.612208 11995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:37:52.613807 11995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:37:52.633339 12010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:37:52.633955 12009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:37:52.640586 12012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:37:52.642542 11995 server_base.cc:1034] running on GCE node
I20260620 02:37:52.643322 11995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:37:52.644604 11995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:37:52.645932 11995 hybrid_clock.cc:648] HybridClock initialized: now 1781923072645877 us; error 57 us; skew 500 ppm
I20260620 02:37:52.646351 11995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:37:52 dist-test-slave-rpg6 krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:37:52.656170 11995 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:37:52.658937 11995 webserver.cc:460] Webserver started at http://127.0.97.2:45953/ using document root <none> and password file <none>
I20260620 02:37:52.659963 11995 fs_manager.cc:359] Metadata directory not provided
I20260620 02:37:52.660125 11995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:37:52.660564 11995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:37:52 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:37:52.681180 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "2c38a1b801f1498aa2031fd7bb276762"
format_stamp: "Formatted at 2026-06-20 02:37:52 on dist-test-slave-rpg6"
server_key: "2916933c2c8be0b46712b0fc9e14f416"
server_key_iv: "e93285a668fb3ffc82fc11d7a88ec3f3"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:52.682803 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2c38a1b801f1498aa2031fd7bb276762"
format_stamp: "Formatted at 2026-06-20 02:37:52 on dist-test-slave-rpg6"
server_key: "2916933c2c8be0b46712b0fc9e14f416"
server_key_iv: "e93285a668fb3ffc82fc11d7a88ec3f3"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:52.706379 11995 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.010s sys 0.011s
I20260620 02:37:52.733201 12019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:37:52.735572 11995 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.008s
I20260620 02:37:52.735944 11995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "2c38a1b801f1498aa2031fd7bb276762"
format_stamp: "Formatted at 2026-06-20 02:37:52 on dist-test-slave-rpg6"
server_key: "2916933c2c8be0b46712b0fc9e14f416"
server_key_iv: "e93285a668fb3ffc82fc11d7a88ec3f3"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:52.736327 11995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:37:52.789202 11995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:37:52.794883 11995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:37:52.795287 11995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:37:52.802817 11995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:37:52.816871 11995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:37:52.817092 11995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 02:37:52.817281 11995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:37:52.817384 11995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:37:52.957392 11995 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36565
I20260620 02:37:52.957432 12131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36565 every 8 connection(s)
I20260620 02:37:52.958947 11995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:37:52.975255 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 11995
I20260620 02:37:52.975451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:37:52.976024 388 external_mini_cluster.cc:1428] Setting key 033cb91606a1ca9e4d389ad6b43ede3c
Jun 20 02:37:52 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:37:52.997571 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:52.962067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44715 (local address 127.0.97.62:43537)
0620 02:37:52.962823 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:52.962827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:52.975619 (+ 12792us) server_negotiation.cc:407] Connection header received
0620 02:37:52.975718 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:52.975724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:52.975844 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:52.975977 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:37:52.979971 (+ 3994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:52.981243 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:52.983802 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:52.984587 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:52.984758 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:52.989078 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:37:52.989087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:37:52.989091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:37:52.989136 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:37:52.992807 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:52.993957 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:52.993993 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:52.993996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:52.994180 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:52.994802 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:52.994808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:52.994810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:52.995184 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:37:52.995320 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:52.996470 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0620 02:37:52.996738 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":529,"threads_started":1}
I20260620 02:37:52.998723 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:52.963122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43537 (local address 127.0.97.2:44715)
0620 02:37:52.972464 (+ 9342us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:52.972529 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 02:37:52.975106 (+ 2577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:37:52.978164 (+ 3058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:37:52.978178 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:37:52.978868 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:37:52.979725 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:52.979753 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:52.981450 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:52.981456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:52.983590 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:52.983608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:52.984897 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:52.984904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:52.985212 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:52.986117 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:37:52.986159 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:37:52.988748 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:37:52.993012 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:37:52.993026 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:37:52.993051 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:37:52.993777 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:37:52.994336 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:37:52.994345 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:37:52.994352 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:37:52.994643 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:37:52.995428 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:37:52.995440 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:37:52.995984 (+ 544us) client_negotiation.cc:769] Sending connection context
0620 02:37:52.996322 (+ 338us) client_negotiation.cc:240] Negotiation successful
0620 02:37:52.996957 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8968,"thread_start_us":8722,"threads_started":1}
I20260620 02:37:53.000649 12132 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43537
I20260620 02:37:53.001003 12132 heartbeater.cc:464] Registering TS with master...
I20260620 02:37:53.001581 11989 heartbeater.cc:503] Master 127.0.97.62:43537 was elected leader, sending a full tablet report...
I20260620 02:37:53.001919 12132 heartbeater.cc:511] Master 127.0.97.62:43537 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:37:53.004232 11776 ts_manager.cc:194] Registered new tserver with Master: 2c38a1b801f1498aa2031fd7bb276762 (127.0.97.2:36565)
I20260620 02:37:53.005103 11776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:37:53.042162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55391/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43537
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:37:53.440030 12138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:37:53.440376 12138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:37:53.440599 12138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:37:53.449405 12138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:37:53.449574 12138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:37:53.449772 12138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:37:53.461059 12138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:55391/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:37:53.462600 12138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:37:53.464102 12138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:37:53.480185 12143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:37:53.480227 12146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:37:53.480280 12144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:37:53.485824 12138 server_base.cc:1034] running on GCE node
I20260620 02:37:53.486536 12138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:37:53.487560 12138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:37:53.488831 12138 hybrid_clock.cc:648] HybridClock initialized: now 1781923073488785 us; error 64 us; skew 500 ppm
I20260620 02:37:53.489262 12138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:37:53 dist-test-slave-rpg6 krb5kdc[11300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:37:53.492530 12138 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:37:53.494163 12138 webserver.cc:460] Webserver started at http://127.0.97.3:37861/ using document root <none> and password file <none>
I20260620 02:37:53.494879 12138 fs_manager.cc:359] Metadata directory not provided
I20260620 02:37:53.495036 12138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:37:53.495324 12138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:37:53 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:37:53.513190 12138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "2093cb905e0647229ed245a266f13c6c"
format_stamp: "Formatted at 2026-06-20 02:37:53 on dist-test-slave-rpg6"
server_key: "78d7e8b20f9ae8f6027c356c51b989ad"
server_key_iv: "df882855312865c202d4e814f22974f2"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:53.514146 12138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2093cb905e0647229ed245a266f13c6c"
format_stamp: "Formatted at 2026-06-20 02:37:53 on dist-test-slave-rpg6"
server_key: "78d7e8b20f9ae8f6027c356c51b989ad"
server_key_iv: "df882855312865c202d4e814f22974f2"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:53.520668 12138 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 02:37:53.538740 12153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:37:53.540531 12138 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:37:53.540697 12138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "2093cb905e0647229ed245a266f13c6c"
format_stamp: "Formatted at 2026-06-20 02:37:53 on dist-test-slave-rpg6"
server_key: "78d7e8b20f9ae8f6027c356c51b989ad"
server_key_iv: "df882855312865c202d4e814f22974f2"
server_key_version: "kuduclusterkey@0"
I20260620 02:37:53.540982 12138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:37:53.568197 12138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:37:53.573243 12138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:37:53.573580 12138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:37:53.574702 12138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:37:53.576297 12138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:37:53.576375 12138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:37:53.576514 12138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:37:53.576563 12138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:37:53.621632 12138 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39769
I20260620 02:37:53.621685 12265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39769 every 8 connection(s)
I20260620 02:37:53.623163 12138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:37:53.632341 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 12138
I20260620 02:37:53.632601 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:37:53.633303 388 external_mini_cluster.cc:1428] Setting key 52fdc29825b0c2dc28561f467b93a387
Jun 20 02:37:53 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:37:53.655395 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.627059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45677 (local address 127.0.97.62:43537)
0620 02:37:53.638053 (+ 10994us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:53.638058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:53.638073 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:37:53.638122 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:53.638126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:53.638248 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:53.638369 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:37:53.640584 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.641468 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.643373 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.643911 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.644015 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.647971 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:37:53.647977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:37:53.647981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:37:53.648017 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:37:53.651456 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:53.652658 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:53.652663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:53.652665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:53.652875 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:53.653446 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:53.653452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:53.653454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:53.653933 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:37:53.654053 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:53.654873 (+ 820us) server_negotiation.cc:299] Negotiation successful
0620 02:37:53.655112 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10879,"thread_start_us":10720,"threads_started":1}
I20260620 02:37:53.656474 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.627793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43537 (local address 127.0.97.3:45677)
0620 02:37:53.635266 (+ 7473us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:53.635334 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 02:37:53.637353 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:37:53.638878 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:37:53.638889 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:37:53.639558 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:37:53.640389 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.640419 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.641628 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.641632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.643212 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.643225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.644206 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.644212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.644431 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.645291 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:37:53.645316 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:37:53.647612 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:37:53.651716 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:37:53.651728 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:37:53.651752 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:37:53.652525 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:37:53.653004 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:37:53.653009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:37:53.653013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:37:53.653302 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:37:53.654061 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:37:53.654068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:37:53.654472 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 02:37:53.654717 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 02:37:53.655146 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7091,"thread_start_us":6846,"threads_started":1}
I20260620 02:37:53.658074 12266 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43537
I20260620 02:37:53.658577 12266 heartbeater.cc:464] Registering TS with master...
I20260620 02:37:53.659402 12266 heartbeater.cc:511] Master 127.0.97.62:43537 requested a full tablet report, sending...
I20260620 02:37:53.661192 11776 ts_manager.cc:194] Registered new tserver with Master: 2093cb905e0647229ed245a266f13c6c (127.0.97.3:39769)
I20260620 02:37:53.662043 11776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45677
I20260620 02:37:53.666453 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:37:53.685745 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.672087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.0.97.62:43537)
0620 02:37:53.672329 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:53.672335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:53.672648 (+ 313us) server_negotiation.cc:407] Connection header received
0620 02:37:53.672853 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:53.672857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:53.672932 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:53.673018 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:37:53.674659 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.675407 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.676839 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.677270 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.677374 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.680085 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:37:53.680091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:37:53.680094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:37:53.680128 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:37:53.682762 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:53.683675 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:53.683680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:53.683682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:53.683801 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:53.684355 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:53.684358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:53.684360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:53.684633 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:37:53.684813 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:53.685222 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 02:37:53.685393 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 02:37:53.690431 11776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:37:53.698112 11776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:37:53.740360 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.713582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33213 (local address 127.0.0.1:59594)
0620 02:37:53.726804 (+ 13222us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:53.726825 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:37:53.727691 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:37:53.728728 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:37:53.728734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:37:53.728761 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:37:53.729312 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.729330 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.731647 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.731656 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.737311 (+ 5655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.737329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.739444 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.739453 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.739642 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.739676 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:37:53.739770 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 02:37:53.739932 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13043,"mutex_wait_us":1,"thread_start_us":18958,"threads_started":1}
I20260620 02:37:53.740471 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.713540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.0.97.2:36565)
0620 02:37:53.727068 (+ 13528us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:53.727077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:53.727109 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:37:53.727224 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:53.727232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:53.727562 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:53.727784 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:37:53.729811 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.731446 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.737343 (+ 5897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.738864 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.739029 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.739278 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:53.739943 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 02:37:53.740125 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13372,"thread_start_us":13259,"threads_started":1}
I20260620 02:37:53.740579 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.714189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.0.97.1:33213)
0620 02:37:53.725355 (+ 11166us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:53.725368 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:53.727630 (+ 2262us) server_negotiation.cc:407] Connection header received
0620 02:37:53.727906 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:53.727914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:53.728193 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:53.728433 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:37:53.729597 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.731446 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.737515 (+ 6069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.739064 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.739195 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.739445 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:53.739911 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 02:37:53.740108 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10957,"thread_start_us":10780,"threads_started":1}
I20260620 02:37:53.741673 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.713500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36565 (local address 127.0.0.1:35026)
0620 02:37:53.725851 (+ 12351us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:53.725894 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 02:37:53.726077 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:37:53.728133 (+ 2056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:37:53.728150 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:37:53.728188 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:37:53.728773 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.728787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.731696 (+ 2909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.731705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.737129 (+ 5424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.737143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.739066 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.739072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.739282 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.739341 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 02:37:53.739475 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 02:37:53.739726 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12102,"thread_start_us":11931,"threads_started":1}
I20260620 02:37:53.745570 12067 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae4d0856c69470b845de78ac2f1f265 (DEFAULT_TABLE table=test-table [id=b4e47c918b1049b7a20d3ef1198f418f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:37:53.745539 11924 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae4d0856c69470b845de78ac2f1f265 (DEFAULT_TABLE table=test-table [id=b4e47c918b1049b7a20d3ef1198f418f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:37:53.746942 11924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae4d0856c69470b845de78ac2f1f265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:37:53.746953 12067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae4d0856c69470b845de78ac2f1f265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:37:53.755503 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.726727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.0.97.3:39769)
0620 02:37:53.734316 (+ 7589us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:53.734331 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:53.735785 (+ 1454us) server_negotiation.cc:407] Connection header received
0620 02:37:53.736037 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:53.736046 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:53.736398 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:53.736635 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:37:53.737384 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.739283 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.751560 (+ 12277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.753786 (+ 2226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.753953 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.754216 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:53.754489 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 02:37:53.754672 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7032,"thread_start_us":6245,"threads_started":1}
I20260620 02:37:53.755522 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.726627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39769 (local address 127.0.0.1:38652)
0620 02:37:53.735609 (+ 8982us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:53.735639 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:37:53.735850 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:37:53.736628 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:37:53.736634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:37:53.736660 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:37:53.737135 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.737143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.739460 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.739468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.742545 (+ 3077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.742564 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.754067 (+ 11503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.754074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.754267 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.754304 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:37:53.754421 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 02:37:53.754579 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8838,"mutex_wait_us":1222,"thread_start_us":7426,"threads_started":1}
I20260620 02:37:53.762387 12286 tablet_bootstrap.cc:492] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: Bootstrap starting.
I20260620 02:37:53.763343 12201 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae4d0856c69470b845de78ac2f1f265 (DEFAULT_TABLE table=test-table [id=b4e47c918b1049b7a20d3ef1198f418f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:37:53.764851 12201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae4d0856c69470b845de78ac2f1f265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:37:53.768185 12286 tablet_bootstrap.cc:654] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: Neither blocks nor log segments found. Creating new log.
I20260620 02:37:53.768884 12285 tablet_bootstrap.cc:492] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: Bootstrap starting.
I20260620 02:37:53.769675 12286 log.cc:824] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: Log is configured to *not* fsync() on all Append() calls
I20260620 02:37:53.772982 12285 tablet_bootstrap.cc:654] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: Neither blocks nor log segments found. Creating new log.
I20260620 02:37:53.773288 12286 tablet_bootstrap.cc:492] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: No bootstrap required, opened a new log
I20260620 02:37:53.773587 12286 ts_tablet_manager.cc:1397] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 02:37:53.774560 12285 log.cc:824] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: Log is configured to *not* fsync() on all Append() calls
I20260620 02:37:53.778494 12285 tablet_bootstrap.cc:492] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: No bootstrap required, opened a new log
I20260620 02:37:53.778810 12285 ts_tablet_manager.cc:1397] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 02:37:53.779894 12289 tablet_bootstrap.cc:492] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: Bootstrap starting.
I20260620 02:37:53.784000 12289 tablet_bootstrap.cc:654] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: Neither blocks nor log segments found. Creating new log.
I20260620 02:37:53.783422 12286 raft_consensus.cc:348] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c38a1b801f1498aa2031fd7bb276762" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36565 } } peers { permanent_uuid: "e7713a95b9544e55a17be8cb3027f9af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33213 } } peers { permanent_uuid: "2093cb905e0647229ed245a266f13c6c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39769 } }
I20260620 02:37:53.784330 12286 raft_consensus.cc:374] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:37:53.784442 12286 raft_consensus.cc:729] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7713a95b9544e55a17be8cb3027f9af, State: Initialized, Role: FOLLOWER
I20260620 02:37:53.785295 12286 consensus_queue.cc:260] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c38a1b801f1498aa2031fd7bb276762" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36565 } } peers { permanent_uuid: "e7713a95b9544e55a17be8cb3027f9af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33213 } } peers { permanent_uuid: "2093cb905e0647229ed245a266f13c6c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39769 } }
I20260620 02:37:53.787909 12289 log.cc:824] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:37:53.788908 12285 raft_consensus.cc:348] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c38a1b801f1498aa2031fd7bb276762" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36565 } } peers { permanent_uuid: "e7713a95b9544e55a17be8cb3027f9af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33213 } } peers { permanent_uuid: "2093cb905e0647229ed245a266f13c6c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39769 } }
I20260620 02:37:53.789400 12285 raft_consensus.cc:374] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:37:53.789500 12285 raft_consensus.cc:729] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c38a1b801f1498aa2031fd7bb276762, State: Initialized, Role: FOLLOWER
I20260620 02:37:53.790285 12285 consensus_queue.cc:260] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c38a1b801f1498aa2031fd7bb276762" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36565 } } peers { permanent_uuid: "e7713a95b9544e55a17be8cb3027f9af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33213 } } peers { permanent_uuid: "2093cb905e0647229ed245a266f13c6c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39769 } }
I20260620 02:37:53.794050 12286 ts_tablet_manager.cc:1428] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260620 02:37:53.797485 12132 heartbeater.cc:503] Master 127.0.97.62:43537 was elected leader, sending a full tablet report...
I20260620 02:37:53.798041 12285 ts_tablet_manager.cc:1428] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260620 02:37:53.798414 12289 tablet_bootstrap.cc:492] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: No bootstrap required, opened a new log
I20260620 02:37:53.798727 12289 ts_tablet_manager.cc:1397] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.010s
I20260620 02:37:53.808396 12289 raft_consensus.cc:348] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c38a1b801f1498aa2031fd7bb276762" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36565 } } peers { permanent_uuid: "e7713a95b9544e55a17be8cb3027f9af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33213 } } peers { permanent_uuid: "2093cb905e0647229ed245a266f13c6c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39769 } }
I20260620 02:37:53.808957 12289 raft_consensus.cc:374] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:37:53.809072 12289 raft_consensus.cc:729] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2093cb905e0647229ed245a266f13c6c, State: Initialized, Role: FOLLOWER
I20260620 02:37:53.809847 12289 consensus_queue.cc:260] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c38a1b801f1498aa2031fd7bb276762" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36565 } } peers { permanent_uuid: "e7713a95b9544e55a17be8cb3027f9af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33213 } } peers { permanent_uuid: "2093cb905e0647229ed245a266f13c6c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39769 } }
I20260620 02:37:53.811404 12266 heartbeater.cc:503] Master 127.0.97.62:43537 was elected leader, sending a full tablet report...
I20260620 02:37:53.811980 12289 ts_tablet_manager.cc:1428] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
W20260620 02:37:53.875555 12267 tablet.cc:2369] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:37:53.954882 11990 tablet.cc:2369] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:37:53.961340 12133 tablet.cc:2369] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:37:53.980813 12293 raft_consensus.cc:482] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:37:53.981139 12293 raft_consensus.cc:504] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c38a1b801f1498aa2031fd7bb276762" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36565 } } peers { permanent_uuid: "e7713a95b9544e55a17be8cb3027f9af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33213 } } peers { permanent_uuid: "2093cb905e0647229ed245a266f13c6c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39769 } }
I20260620 02:37:53.982924 12293 leader_election.cc:288] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c38a1b801f1498aa2031fd7bb276762 (127.0.97.2:36565), e7713a95b9544e55a17be8cb3027f9af (127.0.97.1:33213)
I20260620 02:37:53.990073 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.983560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51635 (local address 127.0.97.1:33213)
0620 02:37:53.983755 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:53.983760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:53.983781 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:37:53.983892 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:53.983897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:53.983955 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:53.984061 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:37:53.984485 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.985976 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.988009 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.989110 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.989234 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.989320 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:53.989689 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 02:37:53.989799 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:37:53.990118 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.983315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33213 (local address 127.0.97.3:51635)
0620 02:37:53.983551 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:53.983573 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:37:53.983736 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:37:53.984082 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:37:53.984086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:37:53.984127 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:37:53.984311 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.984319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.986136 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.986142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.987758 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.987771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.989295 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.989301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.989500 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.989541 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:37:53.989653 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:37:53.989802 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 02:37:53.990935 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.983339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34949 (local address 127.0.97.2:36565)
0620 02:37:53.983555 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:53.983562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:53.984409 (+ 847us) server_negotiation.cc:407] Connection header received
0620 02:37:53.984573 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:53.984576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:53.984648 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:53.984722 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:37:53.985685 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.986720 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.988801 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.989871 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.990000 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.990143 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:53.990532 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 02:37:53.990677 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 02:37:53.991012 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:53.983321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36565 (local address 127.0.97.3:34949)
0620 02:37:53.984298 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0620 02:37:53.984313 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:37:53.984452 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:37:53.985117 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:37:53.985122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:37:53.985147 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:37:53.985493 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.985505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.986895 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.986900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.988615 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:37:53.988630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:37:53.990093 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:37:53.990106 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:37:53.990288 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:53.990325 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:37:53.990471 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 02:37:53.990616 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":186,"thread_start_us":471,"threads_started":1}
I20260620 02:37:53.991400 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae4d0856c69470b845de78ac2f1f265" candidate_uuid: "2093cb905e0647229ed245a266f13c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7713a95b9544e55a17be8cb3027f9af" is_pre_election: true
I20260620 02:37:53.992048 11944 raft_consensus.cc:2454] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2093cb905e0647229ed245a266f13c6c in term 0.
I20260620 02:37:53.992081 12087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae4d0856c69470b845de78ac2f1f265" candidate_uuid: "2093cb905e0647229ed245a266f13c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c38a1b801f1498aa2031fd7bb276762" is_pre_election: true
I20260620 02:37:53.992564 12087 raft_consensus.cc:2454] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2093cb905e0647229ed245a266f13c6c in term 0.
I20260620 02:37:53.992805 12154 leader_election.cc:302] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2093cb905e0647229ed245a266f13c6c, e7713a95b9544e55a17be8cb3027f9af; no voters:
I20260620 02:37:53.993234 12293 raft_consensus.cc:2781] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:37:53.993382 12293 raft_consensus.cc:482] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:37:53.993477 12293 raft_consensus.cc:3037] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:37:53.995662 12293 raft_consensus.cc:504] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c38a1b801f1498aa2031fd7bb276762" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36565 } } peers { permanent_uuid: "e7713a95b9544e55a17be8cb3027f9af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33213 } } peers { permanent_uuid: "2093cb905e0647229ed245a266f13c6c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39769 } }
I20260620 02:37:53.996352 12293 leader_election.cc:288] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [CANDIDATE]: Term 1 election: Requested vote from peers 2c38a1b801f1498aa2031fd7bb276762 (127.0.97.2:36565), e7713a95b9544e55a17be8cb3027f9af (127.0.97.1:33213)
I20260620 02:37:53.996929 12087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae4d0856c69470b845de78ac2f1f265" candidate_uuid: "2093cb905e0647229ed245a266f13c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c38a1b801f1498aa2031fd7bb276762"
I20260620 02:37:53.996922 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae4d0856c69470b845de78ac2f1f265" candidate_uuid: "2093cb905e0647229ed245a266f13c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7713a95b9544e55a17be8cb3027f9af"
I20260620 02:37:53.997150 11944 raft_consensus.cc:3037] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:37:53.997164 12087 raft_consensus.cc:3037] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:37:53.999593 12087 raft_consensus.cc:2454] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2093cb905e0647229ed245a266f13c6c in term 1.
I20260620 02:37:53.999877 11944 raft_consensus.cc:2454] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2093cb905e0647229ed245a266f13c6c in term 1.
I20260620 02:37:54.000151 12157 leader_election.cc:302] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2093cb905e0647229ed245a266f13c6c, 2c38a1b801f1498aa2031fd7bb276762; no voters:
I20260620 02:37:54.000458 12293 raft_consensus.cc:2781] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:37:54.000886 12293 raft_consensus.cc:686] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 1 LEADER]: Becoming Leader. State: Replica: 2093cb905e0647229ed245a266f13c6c, State: Running, Role: LEADER
I20260620 02:37:54.001447 12293 consensus_queue.cc:237] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c38a1b801f1498aa2031fd7bb276762" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36565 } } peers { permanent_uuid: "e7713a95b9544e55a17be8cb3027f9af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33213 } } peers { permanent_uuid: "2093cb905e0647229ed245a266f13c6c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39769 } }
I20260620 02:37:54.006116 11775 catalog_manager.cc:5495] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2093cb905e0647229ed245a266f13c6c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2093cb905e0647229ed245a266f13c6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c38a1b801f1498aa2031fd7bb276762" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7713a95b9544e55a17be8cb3027f9af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2093cb905e0647229ed245a266f13c6c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39769 } health_report { overall_health: HEALTHY } } }
I20260620 02:37:54.032891 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:54.026483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.0.97.3:39769)
0620 02:37:54.026717 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:54.026726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:54.026854 (+ 128us) server_negotiation.cc:407] Connection header received
0620 02:37:54.027124 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:54.027128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:54.027222 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:54.027386 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:37:54.027801 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:54.028619 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:54.030554 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:54.030968 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:54.031111 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:54.031668 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:37:54.032275 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:37:54.032433 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:54.032514 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 02:37:54.032611 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:37:54.041841 11944 raft_consensus.cc:1261] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af [term 1 FOLLOWER]: Refusing update from remote peer 2093cb905e0647229ed245a266f13c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:37:54.041870 12087 raft_consensus.cc:1261] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762 [term 1 FOLLOWER]: Refusing update from remote peer 2093cb905e0647229ed245a266f13c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:37:54.042965 12295 consensus_queue.cc:1035] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7713a95b9544e55a17be8cb3027f9af" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:37:54.043329 12293 consensus_queue.cc:1035] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c38a1b801f1498aa2031fd7bb276762" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:37:54.055706 12298 mvcc.cc:205] Tried to move back new op lower bound from 7298756911264505856 to 7298756911113834496. Current Snapshot: MvccSnapshot[applied={T|T < 7298756911264505856}]
I20260620 02:37:54.073535 11775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:37:54.073853 11775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:37:54.078653 11775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 869ed600e3924b4a8df3c8af9bbf96ee: Sending DeleteTablet for 3 replicas of tablet 2ae4d0856c69470b845de78ac2f1f265
I20260620 02:37:54.080101 11924 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae4d0856c69470b845de78ac2f1f265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:37:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59594
I20260620 02:37:54.080416 12201 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae4d0856c69470b845de78ac2f1f265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:37:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38652
I20260620 02:37:54.080477 12067 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae4d0856c69470b845de78ac2f1f265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:37:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35026
I20260620 02:37:54.082051 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 11856
I20260620 02:37:54.092732 12310 tablet_replica.cc:330] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: stopping tablet replica
I20260620 02:37:54.093622 12310 raft_consensus.cc:2229] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:37:54.094493 12310 raft_consensus.cc:2258] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:37:54.097107 12310 ts_tablet_manager.cc:1905] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:37:54.105275 12310 ts_tablet_manager.cc:1918] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:37:54.105674 12310 log.cc:1196] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/2ae4d0856c69470b845de78ac2f1f265
I20260620 02:37:54.106375 12310 ts_tablet_manager.cc:1939] T 2ae4d0856c69470b845de78ac2f1f265 P e7713a95b9544e55a17be8cb3027f9af: Deleting consensus metadata
I20260620 02:37:54.109694 11760 catalog_manager.cc:4841] TS e7713a95b9544e55a17be8cb3027f9af (127.0.97.1:33213): tablet 2ae4d0856c69470b845de78ac2f1f265 (table test-table [id=b4e47c918b1049b7a20d3ef1198f418f]) successfully deleted
I20260620 02:37:54.110795 12311 tablet_replica.cc:330] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: stopping tablet replica
I20260620 02:37:54.111426 12311 raft_consensus.cc:2229] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:37:54.112341 12311 raft_consensus.cc:2258] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:37:54.113399 12312 tablet_replica.cc:330] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: stopping tablet replica
I20260620 02:37:54.114023 12312 raft_consensus.cc:2229] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:37:54.114706 12312 raft_consensus.cc:2258] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:37:54.116001 12311 ts_tablet_manager.cc:1905] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:37:54 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:37:54.117281 12312 ts_tablet_manager.cc:1905] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:37:54.124788 12311 ts_tablet_manager.cc:1918] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:37:54.125098 12311 log.cc:1196] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/2ae4d0856c69470b845de78ac2f1f265
I20260620 02:37:54.125121 12312 ts_tablet_manager.cc:1918] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:37:54.125324 12312 log.cc:1196] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/2ae4d0856c69470b845de78ac2f1f265
I20260620 02:37:54.125826 12311 ts_tablet_manager.cc:1939] T 2ae4d0856c69470b845de78ac2f1f265 P 2093cb905e0647229ed245a266f13c6c: Deleting consensus metadata
I20260620 02:37:54.125952 12312 ts_tablet_manager.cc:1939] T 2ae4d0856c69470b845de78ac2f1f265 P 2c38a1b801f1498aa2031fd7bb276762: Deleting consensus metadata
I20260620 02:37:54.126732 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:54.083901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59610 (local address 127.0.97.1:33213)
0620 02:37:54.084142 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:54.084151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:54.094951 (+ 10800us) server_negotiation.cc:407] Connection header received
0620 02:37:54.095476 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:54.095485 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:54.095599 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:54.095688 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:37:54.105296 (+ 9608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:54.106485 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:54.109773 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:54.110535 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:54.110665 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:54.117667 (+ 7002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:37:54.117692 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:37:54.117710 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:37:54.117769 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:37:54.122220 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:54.123433 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:54.123445 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:54.123451 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:54.123642 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:54.124244 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:54.124251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:54.124255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:54.124798 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:37:54.125133 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:54.125891 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 02:37:54.126232 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:37:54.128027 11763 catalog_manager.cc:4841] TS 2c38a1b801f1498aa2031fd7bb276762 (127.0.97.2:36565): tablet 2ae4d0856c69470b845de78ac2f1f265 (table test-table [id=b4e47c918b1049b7a20d3ef1198f418f]) successfully deleted
I20260620 02:37:54.128373 11763 catalog_manager.cc:4841] TS 2093cb905e0647229ed245a266f13c6c (127.0.97.3:39769): tablet 2ae4d0856c69470b845de78ac2f1f265 (table test-table [id=b4e47c918b1049b7a20d3ef1198f418f]) successfully deleted
I20260620 02:37:54.128448 11984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:37:54.224433 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 11856
I20260620 02:37:54.250262 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 11995
Jun 20 02:37:54 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:37:54.267311 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:54.251262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.0.97.2:36565)
0620 02:37:54.251458 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:54.251470 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:54.251492 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:37:54.251637 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:54.251643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:54.251709 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:54.251821 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:37:54.253133 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:54.253965 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:54.255308 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:54.255723 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:54.256021 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:54.260593 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:37:54.260617 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:37:54.260642 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:37:54.260689 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:37:54.264160 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:54.264966 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:54.264974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:54.264976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:54.265150 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:54.265532 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:54.265538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:54.265543 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:54.265945 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:37:54.266252 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:54.266635 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 02:37:54.266886 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:37:54.268064 12127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:37:54.372138 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 11995
I20260620 02:37:54.389945 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 12138
Jun 20 02:37:54 dist-test-slave-rpg6 krb5kdc[11300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:37:54.411473 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:37:54.390927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.0.97.3:39769)
0620 02:37:54.391154 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:37:54.391162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:37:54.391184 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:37:54.391251 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:37:54.391256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:37:54.391363 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:37:54.391489 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:37:54.392986 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:54.394208 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:54.396418 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:37:54.397139 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:37:54.397335 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:37:54.402888 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:37:54.402904 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:37:54.402925 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:37:54.402990 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:37:54.407532 (+ 4542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:54.408921 (+ 1389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:54.408932 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:54.408936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:54.409133 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:37:54.409620 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:37:54.409630 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:37:54.409634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:37:54.410144 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:37:54.410468 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 02:37:54.410816 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 02:37:54.411085 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:37:54.412453 12261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:37:54.540226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 12138
I20260620 02:37:54.560438 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 11744
I20260620 02:37:54.561718 11806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:37:54.679272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 11744
I20260620 02:37:54.703617 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 02:37:55.040206 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 02:37:55.040472 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 02:37:55.387633 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 02:37:55.388 UTC [11330] LOG: received smart shutdown request
2026-06-20 02:37:55.392 UTC [11330] LOG: background worker "logical replication launcher" (PID 11339) exited with exit code 1
2026-06-20 02:37:55.392 UTC [11334] LOG: shutting down
2026-06-20 02:37:55.614 UTC [11330] LOG: database system is shut down
2026-06-20T02:37:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55214 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:37:55 dist-test-slave-rpg6 krb5kdc[12326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:37:55 dist-test-slave-rpg6 krb5kdc[12326](info): set up 2 sockets
Jun 20 02:37:55 dist-test-slave-rpg6 krb5kdc[12326](info): commencing operation
krb5kdc: starting...
W20260620 02:37:56.880182 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:37:57 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:37:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 02:37:57.253149 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRSL2lK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/postgres -l logfile start
2026-06-20 02:38:00.308 UTC [12356] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 02:38:00.308 UTC [12356] LOG: listening on IPv4 address "127.0.97.20", port 49197
2026-06-20 02:38:00.312 UTC [12356] LOG: listening on Unix socket "/tmp/.s.PGSQL.49197"
2026-06-20 02:38:00.334 UTC [12359] LOG: database system was shut down at 2026-06-20 02:37:58 UTC
2026-06-20 02:38:00.338 UTC [12356] LOG: database system is ready to accept connections
I20260620 02:38:01.327804 388 mini_postgres.cc:96] Postgres bound to 49197
2026-06-20 02:38:01.345 UTC [12367] FATAL: database "slave" does not exist
127.0.97.20:49197 - accepting connections
I20260620 02:38:01.346271 388 mini_ranger.cc:161] Starting Ranger...
I20260620 02:38:01.374109 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 02:38:02.033670 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 02:38:02.033772 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-admin
2026-06-20 02:38:02,345 [I] DB FLAVOR :POSTGRES
2026-06-20 02:38:02,345 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:38:02,346 [I] Checking connection..
2026-06-20 02:38:02,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 02:38:02,621 [I] Checking connection passed.
2026-06-20 02:38:02,622 [I] --------- Verifying version history table ---------
2026-06-20 02:38:02,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:38:02,888 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:38:02,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:38:03,145 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:38:03,145 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 02:38:03,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 02:38:03.421 UTC [12431] WARNING: there is no transaction in progress
2026-06-20 02:38:03,428 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 02:38:03,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:38:03,715 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 02:38:03,715 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:38:03,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 02:38:03,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 02:38:04,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rpg6.c.gcp-upstream.internal','N') ;"
2026-06-20 02:38:04,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:38:04,784 [I] Table x_portal_user does not exist in database ranger
2026-06-20 02:38:04,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:38:05,042 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 02:38:05,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:38:05,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:38:05,567 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 02:38:05,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 02:38:06.731 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.751 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.772 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.793 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.816 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.836 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.902 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.911 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.930 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.942 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.954 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.970 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.981 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.990 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:06.999 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:07.008 UTC [12539] WARNING: there is no transaction in progress
2026-06-20 02:38:07,870 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 02:38:07,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 02:38:08,140 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 02:38:08,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:38:08,420 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 02:38:08,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:38:08,672 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 02:38:08,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:38:08,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:38:09,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rpg6.c.gcp-upstream.internal';"
2026-06-20 02:38:09,483 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 02:38:09,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 02:38:09,775 [I] DB_PATCHES have already been applied
I20260620 02:38:09.783723 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRSL2lK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 02:38:09.783978 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 02:38:09.801919 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:58987
Jun 20, 2026 2:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:38:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 02:38:10.969343 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.623s user 0.000s sys 0.047s
Jun 20, 2026 2:38:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:38:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:38:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:38:32 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:38:33 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 02:38:33.727461 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 02:38:33 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923113, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 02:38:33.931994 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 02:38:33.956662 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 02:38:35.166301 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 02:38:35.166419 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-kms
I20260620 02:38:35.166467 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:49197
2026-06-20 02:38:35,550 [I] DB FLAVOR :POSTGRES
2026-06-20 02:38:35,550 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:38:35,550 [I] Checking connection
2026-06-20 02:38:35,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 02:38:35,835 [I] connection success
2026-06-20 02:38:35,835 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 02:38:35,835 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 02:38:35,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 02:38:36,118 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 02:38:36,119 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:38:36,119 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 02:38:36,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 02:38:36,438 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 02:38:37.031996 388 mini_ranger_kms.cc:310] Created kms service
I20260620 02:38:37.276671 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 02:38:37.332619 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 02:38:37 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923113, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:38:37.851950 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 02:38:37.852293 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRSL2lK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRSL2lK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 02:38:37.852351 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 02:38:37.871451 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12720
I20260620 02:38:37.871591 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:46049
Jun 20, 2026 2:38:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:38:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:38:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:38:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:38:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:38:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 02:38:40.196674 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.265s user 0.092s sys 0.312s
I20260620 02:38:40.196964 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 02:38:40.197153 388 mini_ranger_kms.cc:401] 127.0.97.20:46049/kms/v1/keys
Jun 20 02:38:40 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781923120, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 02:38:40 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923120, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:38:41 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923120, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:38:43 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:38:43.756486 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33285
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33285
--ranger_config_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46049/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:38:44.122179 12770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:38:44.122493 12770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:38:44.122659 12770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:38:44.130479 12770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:38:44.130611 12770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:38:44.130685 12770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:38:44.130770 12770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:38:44.130854 12770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:38:44.141957 12770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:46049/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33285
--ranger_config_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:38:44.143401 12770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:38:44.144922 12770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:38:44.158327 12775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:44.158316 12776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:44.158392 12778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:44.159127 12770 server_base.cc:1034] running on GCE node
I20260620 02:38:44.160105 12770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:38:44.161759 12770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:38:44.163017 12770 hybrid_clock.cc:648] HybridClock initialized: now 1781923124162965 us; error 58 us; skew 500 ppm
I20260620 02:38:44.163367 12770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:38:44 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:44.171185 12770 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:38:44.173243 12770 webserver.cc:460] Webserver started at http://127.0.97.62:33755/ using document root <none> and password file <none>
I20260620 02:38:44.173990 12770 fs_manager.cc:359] Metadata directory not provided
I20260620 02:38:44.174157 12770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:38:44.174523 12770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:38:44 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:38:44.235812 12770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "c78433e90108484693583312f062934a"
format_stamp: "Formatted at 2026-06-20 02:38:44 on dist-test-slave-rpg6"
server_key: "392a7ff9ea66081835eda45eb30673e6"
server_key_iv: "9543f23c507d737f6db70c9448b0347c"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:44.237406 12770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "c78433e90108484693583312f062934a"
format_stamp: "Formatted at 2026-06-20 02:38:44 on dist-test-slave-rpg6"
server_key: "392a7ff9ea66081835eda45eb30673e6"
server_key_iv: "9543f23c507d737f6db70c9448b0347c"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:44.245641 12770 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 02:38:44.268929 12785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:44.271294 12770 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:38:44.271539 12770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "c78433e90108484693583312f062934a"
format_stamp: "Formatted at 2026-06-20 02:38:44 on dist-test-slave-rpg6"
server_key: "392a7ff9ea66081835eda45eb30673e6"
server_key_iv: "9543f23c507d737f6db70c9448b0347c"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:44.271880 12770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:38:44.294623 12770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:38:44.300390 12770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:38:44.300773 12770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:38:44.324707 12770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33285
I20260620 02:38:44.324735 12836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33285 every 8 connection(s)
I20260620 02:38:44.326495 12770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:38:44.331302 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:38:44.335073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 12770
I20260620 02:38:44.335317 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:38:44.336169 388 external_mini_cluster.cc:1428] Setting key 130055d3c04c22321fc78e74992c59cc
I20260620 02:38:44.348568 12837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a: Bootstrap starting.
I20260620 02:38:44.354812 12837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a: Neither blocks nor log segments found. Creating new log.
I20260620 02:38:44.356202 12837 log.cc:824] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:38:44.360031 12837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a: No bootstrap required, opened a new log
Jun 20 02:38:44 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:38:44.370723 12837 raft_consensus.cc:348] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260620 02:38:44.371274 12837 raft_consensus.cc:374] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:38:44.371411 12837 raft_consensus.cc:729] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78433e90108484693583312f062934a, State: Initialized, Role: FOLLOWER
I20260620 02:38:44.372546 12837 consensus_queue.cc:260] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260620 02:38:44.372900 12837 raft_consensus.cc:388] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:38:44.373101 12837 raft_consensus.cc:482] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:38:44.373338 12837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:38:44.376602 12837 raft_consensus.cc:504] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260620 02:38:44.377362 12837 leader_election.cc:302] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c78433e90108484693583312f062934a; no voters:
I20260620 02:38:44.378122 12837 leader_election.cc:288] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:38:44.378455 12842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:38:44.382638 12842 raft_consensus.cc:686] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 1 LEADER]: Becoming Leader. State: Replica: c78433e90108484693583312f062934a, State: Running, Role: LEADER
I20260620 02:38:44.382922 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:44.338547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.0.97.62:33285)
0620 02:38:44.353334 (+ 14787us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:44.353351 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:44.353406 (+ 55us) server_negotiation.cc:407] Connection header received
0620 02:38:44.356019 (+ 2613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:44.356056 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:44.356634 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:44.357220 (+ 586us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:44.359034 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:44.361161 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:44.363215 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:44.364528 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:44.364748 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:44.370173 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:44.370188 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:44.370217 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:44.370283 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:44.375049 (+ 4766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:44.376103 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:44.376114 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:44.376124 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:44.376354 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:44.376888 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:44.376893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:44.376897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:44.377818 (+ 921us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:44.378165 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:44.380236 (+ 2071us) server_negotiation.cc:299] Negotiation successful
0620 02:38:44.380900 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14515,"thread_start_us":14264,"threads_started":1}
I20260620 02:38:44.383430 12842 consensus_queue.cc:237] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260620 02:38:44.387466 12837 sys_catalog.cc:564] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:38:44.389652 12844 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c78433e90108484693583312f062934a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } } }
I20260620 02:38:44.390097 12844 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [sys.catalog]: This master's current role is: LEADER
I20260620 02:38:44.392577 12843 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c78433e90108484693583312f062934a. Latest consensus state: current_term: 1 leader_uuid: "c78433e90108484693583312f062934a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } } }
I20260620 02:38:44.393190 12843 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [sys.catalog]: This master's current role is: LEADER
I20260620 02:38:44.401978 12847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:38:44.409428 12847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:38:44.427431 12847 catalog_manager.cc:1347] Generated new cluster ID: 0a1a9563bb9d4e4181d9c683ebb0e4ad
I20260620 02:38:44.427716 12847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:38:44.444370 12847 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:38:44.446328 12847 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:38:44.457401 12847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a: Generated new TSK 0
I20260620 02:38:44.457643 12837 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 02:38:44.458689 12847 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:38:45.331333 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46049/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:38:45 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781923125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 02:38:45.786650 12882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:38:45.787258 12882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:38:45.787472 12882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:38:45.800201 12882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:38:45.800369 12882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:38:45.800621 12882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:38:45.817423 12882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:46049/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:38:45.819407 12882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:38:45.821408 12882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:38:45.841156 12892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:45.842123 12893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:45.851218 12882 server_base.cc:1034] running on GCE node
W20260620 02:38:45.851660 12895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:45.852614 12882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:38:45.854009 12882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:38:45.855324 12882 hybrid_clock.cc:648] HybridClock initialized: now 1781923125855254 us; error 61 us; skew 500 ppm
I20260620 02:38:45.855749 12882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:38:45 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:45.860577 12882 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:38:45.864248 12882 webserver.cc:460] Webserver started at http://127.0.97.1:38595/ using document root <none> and password file <none>
I20260620 02:38:45.865366 12882 fs_manager.cc:359] Metadata directory not provided
I20260620 02:38:45.865542 12882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:38:45.865988 12882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:38:45 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:38:45.887679 12882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "02112b4f3b654539bef7497035387aa7"
format_stamp: "Formatted at 2026-06-20 02:38:45 on dist-test-slave-rpg6"
server_key: "300945d368b2261659b1dc6d10ac0e29"
server_key_iv: "0b3dae47e6bbc56d2faedfb0da1a39fc"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:45.890826 12882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "02112b4f3b654539bef7497035387aa7"
format_stamp: "Formatted at 2026-06-20 02:38:45 on dist-test-slave-rpg6"
server_key: "300945d368b2261659b1dc6d10ac0e29"
server_key_iv: "0b3dae47e6bbc56d2faedfb0da1a39fc"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:45.906034 12882 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.011s sys 0.001s
I20260620 02:38:45.928480 12902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:45.930361 12882 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260620 02:38:45.930583 12882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "02112b4f3b654539bef7497035387aa7"
format_stamp: "Formatted at 2026-06-20 02:38:45 on dist-test-slave-rpg6"
server_key: "300945d368b2261659b1dc6d10ac0e29"
server_key_iv: "0b3dae47e6bbc56d2faedfb0da1a39fc"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:45.930850 12882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:38:46.035337 12882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:38:46.042392 12882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:38:46.042874 12882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:38:46.044323 12882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:38:46.046669 12882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:38:46.046881 12882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 02:38:46.047119 12882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:38:46.047200 12882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:46.108868 12882 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36999
I20260620 02:38:46.108899 13014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36999 every 8 connection(s)
I20260620 02:38:46.110713 12882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:38:46.112488 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 12882
I20260620 02:38:46.112702 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:38:46.113535 388 external_mini_cluster.cc:1428] Setting key 1a236ff942980c3c739bf6473a862403
Jun 20 02:38:46 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:38:46.164852 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:46.115500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38379 (local address 127.0.97.62:33285)
0620 02:38:46.129415 (+ 13915us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:46.129424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:46.129457 (+ 33us) server_negotiation.cc:407] Connection header received
0620 02:38:46.131329 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:46.131339 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:46.131460 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:46.131595 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:46.134189 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:46.135674 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:46.138849 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:46.139697 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:46.139863 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:46.147817 (+ 7954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:46.147863 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:46.147869 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:46.147923 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:46.151928 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:46.153930 (+ 2002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:46.153938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:46.153941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:46.154145 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:46.154890 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:46.154896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:46.154899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:46.155333 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:46.155484 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:46.164106 (+ 8622us) server_negotiation.cc:299] Negotiation successful
0620 02:38:46.164414 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13737,"thread_start_us":13590,"threads_started":1}
I20260620 02:38:46.167460 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:46.116449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33285 (local address 127.0.97.1:38379)
0620 02:38:46.126212 (+ 9763us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:46.126276 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:46.130764 (+ 4488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:46.132163 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:46.132181 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:46.132954 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:46.133906 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:46.133940 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:46.136040 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:46.136048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:46.138592 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:46.138628 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:46.140332 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:46.140343 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:46.140762 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:46.144174 (+ 3412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:46.144215 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:46.147436 (+ 3221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:46.152192 (+ 4756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:46.152214 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:46.152238 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:46.153712 (+ 1474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:46.154342 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:46.154348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:46.154352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:46.154694 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:46.162787 (+ 8093us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:46.162800 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:46.163429 (+ 629us) client_negotiation.cc:769] Sending connection context
0620 02:38:46.165167 (+ 1738us) client_negotiation.cc:240] Negotiation successful
0620 02:38:46.165890 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9399,"thread_start_us":11236,"threads_started":1}
I20260620 02:38:46.170159 13015 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33285
I20260620 02:38:46.170574 13015 heartbeater.cc:464] Registering TS with master...
I20260620 02:38:46.171506 13015 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
I20260620 02:38:46.174294 12802 ts_manager.cc:194] Registered new tserver with Master: 02112b4f3b654539bef7497035387aa7 (127.0.97.1:36999)
I20260620 02:38:46.176471 12802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38379
Jun 20 02:38:46 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:38:46.204459 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46049/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:38:46 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 02:38:46.767565 13021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:38:46.768153 13021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:38:46.768468 13021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:38:46.781349 13021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:38:46.781625 13021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:38:46.781859 13021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:38:46.799047 13021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:46049/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:38:46.801295 13021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:38:46.803296 13021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:38:46.855402 13029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:46.856113 13028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:46.861790 13031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:46.862937 13021 server_base.cc:1034] running on GCE node
I20260620 02:38:46.863926 13021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:38:46.871611 13021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:38:46.873167 13021 hybrid_clock.cc:648] HybridClock initialized: now 1781923126873123 us; error 38 us; skew 500 ppm
I20260620 02:38:46.873746 13021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:38:46 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:46.879510 13021 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:38:46.917253 13021 webserver.cc:460] Webserver started at http://127.0.97.2:35169/ using document root <none> and password file <none>
I20260620 02:38:46.918560 13021 fs_manager.cc:359] Metadata directory not provided
I20260620 02:38:46.918845 13021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:38:46.919334 13021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:38:46 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:38:46.960891 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "5a2f81bb775d42b48400dd015dfd8802"
format_stamp: "Formatted at 2026-06-20 02:38:46 on dist-test-slave-rpg6"
server_key: "61ba3b59728669e7aa1f5536b74ced30"
server_key_iv: "c91e18903455a88ecd4b5850477b81b5"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:46.978865 13021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5a2f81bb775d42b48400dd015dfd8802"
format_stamp: "Formatted at 2026-06-20 02:38:46 on dist-test-slave-rpg6"
server_key: "61ba3b59728669e7aa1f5536b74ced30"
server_key_iv: "c91e18903455a88ecd4b5850477b81b5"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:47.000424 13021 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.006s sys 0.001s
I20260620 02:38:47.040756 13040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:47.043655 13021 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.002s sys 0.016s
I20260620 02:38:47.043967 13021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "5a2f81bb775d42b48400dd015dfd8802"
format_stamp: "Formatted at 2026-06-20 02:38:46 on dist-test-slave-rpg6"
server_key: "61ba3b59728669e7aa1f5536b74ced30"
server_key_iv: "c91e18903455a88ecd4b5850477b81b5"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:47.044289 13021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:38:47.123754 13021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:38:47.128669 13021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:38:47.129093 13021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:38:47.130517 13021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:38:47.132164 13021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:38:47.132272 13021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:47.132439 13021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:38:47.132516 13021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:47.179128 13021 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34903
I20260620 02:38:47.179152 13157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34903 every 8 connection(s)
I20260620 02:38:47.180648 13021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:38:47.181319 13015 heartbeater.cc:503] Master 127.0.97.62:33285 was elected leader, sending a full tablet report...
I20260620 02:38:47.189410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 13021
I20260620 02:38:47.189630 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:38:47.190390 388 external_mini_cluster.cc:1428] Setting key 4b90117358ac43cd80357f1c9d66c71a
Jun 20 02:38:47 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:38:47.230279 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:47.187532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33285 (local address 127.0.97.2:59819)
0620 02:38:47.200635 (+ 13103us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:47.200702 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:47.205210 (+ 4508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:47.208359 (+ 3149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:47.208375 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:47.209177 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:47.210124 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:47.210157 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:47.211961 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:47.211967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:47.214127 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:47.214149 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:47.215721 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:47.215730 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:47.216081 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:47.217241 (+ 1160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:47.217274 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:47.220364 (+ 3090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:47.224833 (+ 4469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:47.224844 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:47.224867 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:47.225526 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:47.226021 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:47.226026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:47.226030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:47.226287 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:47.227008 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:47.227017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:47.227479 (+ 462us) client_negotiation.cc:769] Sending connection context
0620 02:38:47.227802 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 02:38:47.228448 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12756,"thread_start_us":12462,"threads_started":1}
I20260620 02:38:47.236306 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:47.186900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59819 (local address 127.0.97.62:33285)
0620 02:38:47.192018 (+ 5118us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:47.192025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:47.202505 (+ 10480us) server_negotiation.cc:407] Connection header received
0620 02:38:47.205654 (+ 3149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:47.205661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:47.205767 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:47.205889 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:47.210476 (+ 4587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:47.211646 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:47.214363 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:47.215041 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:47.215150 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:47.220703 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:47.220711 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:47.220716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:47.220769 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:47.224575 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:47.225705 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:47.225712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:47.225716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:47.225882 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:47.226418 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:47.226424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:47.226428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:47.226851 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:47.227020 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:47.234904 (+ 7884us) server_negotiation.cc:299] Negotiation successful
0620 02:38:47.235908 (+ 1004us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4967,"thread_start_us":4758,"threads_started":1}
I20260620 02:38:47.238564 13158 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33285
I20260620 02:38:47.239018 13158 heartbeater.cc:464] Registering TS with master...
I20260620 02:38:47.240437 13158 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
I20260620 02:38:47.242470 12802 ts_manager.cc:194] Registered new tserver with Master: 5a2f81bb775d42b48400dd015dfd8802 (127.0.97.2:34903)
I20260620 02:38:47.243605 12802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:38:47.269605 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46049/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:38:47.699636 13164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:38:47.700023 13164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:38:47.700199 13164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:38:47.708082 13164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:38:47.708218 13164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:38:47.708420 13164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:38:47.720129 13164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:46049/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:38:47.721585 13164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:38:47.723474 13164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:38:47.739356 13170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:47.739341 13172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:47.740381 13169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:47.740779 13164 server_base.cc:1034] running on GCE node
I20260620 02:38:47.741791 13164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:38:47.743047 13164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:38:47.744346 13164 hybrid_clock.cc:648] HybridClock initialized: now 1781923127744243 us; error 90 us; skew 500 ppm
I20260620 02:38:47.744760 13164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:38:47 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:47.748711 13164 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:38:47.750722 13164 webserver.cc:460] Webserver started at http://127.0.97.3:46205/ using document root <none> and password file <none>
I20260620 02:38:47.751518 13164 fs_manager.cc:359] Metadata directory not provided
I20260620 02:38:47.751724 13164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:38:47.752121 13164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:38:47 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:38:47.772233 13164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "70d920ad3c9a41f09d48dd9ae0218f17"
format_stamp: "Formatted at 2026-06-20 02:38:47 on dist-test-slave-rpg6"
server_key: "92904083b3f11fc746afdfcc0cc8075e"
server_key_iv: "a4d2a6d2d147dd9ed47c55278761d535"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:47.773212 13164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "70d920ad3c9a41f09d48dd9ae0218f17"
format_stamp: "Formatted at 2026-06-20 02:38:47 on dist-test-slave-rpg6"
server_key: "92904083b3f11fc746afdfcc0cc8075e"
server_key_iv: "a4d2a6d2d147dd9ed47c55278761d535"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:47.780066 13164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:38:47.800359 13179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:47.802737 13164 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:38:47.803030 13164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "70d920ad3c9a41f09d48dd9ae0218f17"
format_stamp: "Formatted at 2026-06-20 02:38:47 on dist-test-slave-rpg6"
server_key: "92904083b3f11fc746afdfcc0cc8075e"
server_key_iv: "a4d2a6d2d147dd9ed47c55278761d535"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:47.803349 13164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:38:47.826720 13164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:38:47.832803 13164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:38:47.833367 13164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:38:47.834667 13164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:38:47.836516 13164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:38:47.836622 13164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:47.836764 13164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:38:47.836814 13164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:47.878609 13164 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34785
I20260620 02:38:47.878641 13291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34785 every 8 connection(s)
I20260620 02:38:47.880066 13164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:38:47.887131 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 13164
I20260620 02:38:47.887394 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:38:47.888073 388 external_mini_cluster.cc:1428] Setting key b8ba6aa999db35ed6c85f5e626e22d74
Jun 20 02:38:47 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:38:47.904033 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:47.883003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40925 (local address 127.0.97.62:33285)
0620 02:38:47.883539 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:47.883544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:47.884698 (+ 1154us) server_negotiation.cc:407] Connection header received
0620 02:38:47.886712 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:47.886718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:47.886848 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:47.886977 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:47.889002 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:47.889790 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:47.891485 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:47.892140 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:47.892283 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:47.896964 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:47.896971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:47.896975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:47.897022 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:47.900235 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:47.901278 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:47.901290 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:47.901292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:47.901442 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:47.901970 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:47.901977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:47.901986 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:47.902315 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:47.902485 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:47.903289 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 02:38:47.903548 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
I20260620 02:38:47.905293 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:47.883511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33285 (local address 127.0.97.3:40925)
0620 02:38:47.884475 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:47.884525 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:47.886352 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:47.887331 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:47.887347 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:47.887993 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:47.888799 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:47.888822 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:47.889967 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:47.889980 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:47.891315 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:47.891328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:47.892386 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:47.892396 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:47.892679 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:47.893510 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:47.893536 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:47.896526 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:47.900508 (+ 3982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:47.900528 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:47.900547 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:47.901105 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:47.901583 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:47.901588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:47.901592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:47.901832 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:47.902454 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:47.902463 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:47.902876 (+ 413us) client_negotiation.cc:769] Sending connection context
0620 02:38:47.903164 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 02:38:47.903886 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":527,"threads_started":1}
I20260620 02:38:47.907927 13292 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33285
I20260620 02:38:47.908388 13292 heartbeater.cc:464] Registering TS with master...
I20260620 02:38:47.909238 13292 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
I20260620 02:38:47.911763 12802 ts_manager.cc:194] Registered new tserver with Master: 70d920ad3c9a41f09d48dd9ae0218f17 (127.0.97.3:34785)
I20260620 02:38:47.913121 12802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40925
I20260620 02:38:47.916749 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:38:47.918567 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 12882
Jun 20 02:38:47 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:38:47.938403 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:47.919725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.0.97.1:36999)
0620 02:38:47.920405 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:47.920412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:47.920440 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:38:47.920498 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:47.920504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:47.920725 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:47.920857 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:47.922528 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:47.923597 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:47.925472 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:47.926405 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:47.926614 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:47.931336 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:47.931361 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:47.931385 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:47.931444 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:47.934815 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:47.935789 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:47.935797 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:47.935800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:47.936117 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:47.936631 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:47.936635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:47.936637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:47.937011 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:47.937272 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:47.937731 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 02:38:47.938032 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":349,"threads_started":1}
I20260620 02:38:47.939889 13010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:38:48.028942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 12882
I20260620 02:38:48.045428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 13021
Jun 20 02:38:48 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:38:48.066349 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:48.046365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.0.97.2:34903)
0620 02:38:48.046976 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:48.046983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:48.047014 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:38:48.047074 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:48.047082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:48.047320 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:48.047454 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:48.048911 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:48.050109 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:48.052274 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:48.052899 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:48.053020 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:48.057963 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:48.058002 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:48.058063 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:48.058121 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:48.062199 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:48.063245 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:48.063254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:48.063257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:48.063453 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:48.064312 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:48.064317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:48.064320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:48.064877 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:48.065158 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:48.065694 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 02:38:48.066011 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":347,"threads_started":1}
I20260620 02:38:48.067775 13153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:38:48.159412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 13021
I20260620 02:38:48.180531 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 13164
Jun 20 02:38:48 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:38:48.203337 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:48.182118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.0.97.3:34785)
0620 02:38:48.183209 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:48.183218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:48.183261 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:38:48.183395 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:48.183404 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:48.183724 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:48.184049 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:48.185925 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:48.187049 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:48.189593 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:48.190332 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:48.190534 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:48.196050 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:48.196078 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:48.196109 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:48.196180 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:48.200052 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:48.201001 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:48.201009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:48.201011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:48.201165 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:48.201575 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:48.201582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:48.201587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:48.201978 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:48.202334 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:48.202658 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 02:38:48.202946 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":721,"threads_started":1}
I20260620 02:38:48.204770 13287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:38:48.294903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 13164
I20260620 02:38:48.312342 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 12770
I20260620 02:38:48.313727 12832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:38:48.425025 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 12770
I20260620 02:38:48.444231 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33285
--webserver_interface=127.0.97.62
--webserver_port=33755
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33285
--ranger_config_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46049/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46049/kms with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:38:48.801281 13304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:38:48.801628 13304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:38:48.801805 13304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:38:48.809624 13304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:38:48.809769 13304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:38:48.809859 13304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:38:48.809949 13304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:38:48.810016 13304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:38:48.820988 13304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46049/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33285
--ranger_config_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:38:48.822562 13304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:38:48.824115 13304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:38:48.837021 13312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:48.837041 13310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:48.837026 13309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:48.838014 13304 server_base.cc:1034] running on GCE node
I20260620 02:38:48.839022 13304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:38:48.840754 13304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:38:48.842037 13304 hybrid_clock.cc:648] HybridClock initialized: now 1781923128842005 us; error 53 us; skew 500 ppm
I20260620 02:38:48.842521 13304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:38:48 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:48.846419 13304 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:38:48.848289 13304 webserver.cc:460] Webserver started at http://127.0.97.62:33755/ using document root <none> and password file <none>
I20260620 02:38:48.849037 13304 fs_manager.cc:359] Metadata directory not provided
I20260620 02:38:48.849205 13304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:38:48 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:38:48.940311 13304 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:38:48.943959 13320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:48.946370 13304 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:38:48.946635 13304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "c78433e90108484693583312f062934a"
format_stamp: "Formatted at 2026-06-20 02:38:44 on dist-test-slave-rpg6"
server_key: "392a7ff9ea66081835eda45eb30673e6"
server_key_iv: "9543f23c507d737f6db70c9448b0347c"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:48.947772 13304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:38:48.967090 13304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:38:48.972050 13304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:38:48.972405 13304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:38:48.995395 13371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33285 every 8 connection(s)
I20260620 02:38:48.995455 13304 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33285
I20260620 02:38:48.997221 13304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:38:49.002590 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 13304
I20260620 02:38:49.003446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46049/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36999
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33285
--webserver_port=38595
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46049/kms with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:38:49.004709 13372 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:38:49.008263 13372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a: Bootstrap starting.
I20260620 02:38:49.028313 13372 log.cc:824] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:38:49.038213 13372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:38:49.038843 13372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a: Bootstrap complete.
I20260620 02:38:49.048448 13372 raft_consensus.cc:348] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260620 02:38:49.049433 13372 raft_consensus.cc:729] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78433e90108484693583312f062934a, State: Initialized, Role: FOLLOWER
I20260620 02:38:49.050262 13372 consensus_queue.cc:260] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260620 02:38:49.050588 13372 raft_consensus.cc:388] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:38:49.050741 13372 raft_consensus.cc:482] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:38:49.050966 13372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:38:49.055362 13372 raft_consensus.cc:504] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260620 02:38:49.056006 13372 leader_election.cc:302] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c78433e90108484693583312f062934a; no voters:
I20260620 02:38:49.056677 13372 leader_election.cc:288] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:38:49.056846 13376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:38:49.058081 13376 raft_consensus.cc:686] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [term 2 LEADER]: Becoming Leader. State: Replica: c78433e90108484693583312f062934a, State: Running, Role: LEADER
I20260620 02:38:49.058789 13376 consensus_queue.cc:237] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } }
I20260620 02:38:49.059140 13372 sys_catalog.cc:564] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:38:49.061530 13378 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c78433e90108484693583312f062934a. Latest consensus state: current_term: 2 leader_uuid: "c78433e90108484693583312f062934a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } } }
I20260620 02:38:49.061935 13378 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [sys.catalog]: This master's current role is: LEADER
I20260620 02:38:49.061653 13377 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c78433e90108484693583312f062934a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78433e90108484693583312f062934a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33285 } } }
I20260620 02:38:49.062500 13377 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a [sys.catalog]: This master's current role is: LEADER
I20260620 02:38:49.067224 13383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:38:49.074285 13383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:38:49.076365 13383 catalog_manager.cc:1259] Loaded cluster ID: 0a1a9563bb9d4e4181d9c683ebb0e4ad
I20260620 02:38:49.076608 13383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:38:49.082199 13383 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:38:49.084448 13383 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a: Loaded TSK: 0
I20260620 02:38:49.086694 13383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:38:49.100705 13372 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 02:38:49.497656 13374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:38:49.498100 13374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:38:49.498334 13374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:38:49.510986 13374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:38:49.511222 13374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:38:49.511471 13374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:38:49.528156 13374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46049/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38595
--tserver_master_addrs=127.0.97.62:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:38:49.530205 13374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:38:49.532179 13374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:38:49.570047 13410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:49.571807 13409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:49.572911 13412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:49.577104 13374 server_base.cc:1034] running on GCE node
I20260620 02:38:49.578109 13374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:38:49.579675 13374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:38:49.583210 13374 hybrid_clock.cc:648] HybridClock initialized: now 1781923129583108 us; error 79 us; skew 500 ppm
I20260620 02:38:49.583811 13374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:38:49 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:49.595199 13374 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:38:49.626147 13374 webserver.cc:460] Webserver started at http://127.0.97.1:38595/ using document root <none> and password file <none>
I20260620 02:38:49.627379 13374 fs_manager.cc:359] Metadata directory not provided
I20260620 02:38:49.627775 13374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:38:49 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:38:49.676577 13374 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.008s sys 0.004s
I20260620 02:38:49.693409 13421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:49.695781 13374 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.009s sys 0.007s
I20260620 02:38:49.696108 13374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "02112b4f3b654539bef7497035387aa7"
format_stamp: "Formatted at 2026-06-20 02:38:45 on dist-test-slave-rpg6"
server_key: "300945d368b2261659b1dc6d10ac0e29"
server_key_iv: "0b3dae47e6bbc56d2faedfb0da1a39fc"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:49.697800 13374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:38:49.715286 13374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:38:49.720474 13374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:38:49.720906 13374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:38:49.722223 13374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:38:49.724243 13374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:38:49.724432 13374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:49.724645 13374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:38:49.724785 13374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:49.775205 13374 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36999
I20260620 02:38:49.775290 13533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36999 every 8 connection(s)
I20260620 02:38:49.777415 13374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:38:49.786947 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 13374
I20260620 02:38:49.788056 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46049/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34903
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33285
--webserver_port=35169
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46049/kms with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:38:49 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:38:49.841758 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:49.782370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38855 (local address 127.0.97.62:33285)
0620 02:38:49.802763 (+ 20393us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:49.802797 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:49.809218 (+ 6421us) server_negotiation.cc:407] Connection header received
0620 02:38:49.811324 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:49.811353 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:49.812015 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:49.812534 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:49.815635 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:49.818881 (+ 3246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:49.822601 (+ 3720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:49.824121 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:49.824370 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:49.830666 (+ 6296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:49.830688 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:49.830713 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:49.830790 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:49.835502 (+ 4712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:49.836801 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:49.836808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:49.836815 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:49.837040 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:49.837660 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:49.837670 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:49.837673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:49.838317 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:49.838520 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:49.839684 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0620 02:38:49.840240 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19992,"thread_start_us":19699,"threads_started":1}
I20260620 02:38:49.844844 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:49.781475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33285 (local address 127.0.97.1:38855)
0620 02:38:49.805460 (+ 23985us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:49.805542 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:49.810308 (+ 4766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:49.813534 (+ 3226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:49.813550 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:49.814324 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:49.815303 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:49.815338 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:49.819264 (+ 3926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:49.819276 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:49.822284 (+ 3008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:49.822319 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:49.824647 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:49.824657 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:49.824924 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:49.825815 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:49.825857 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:49.830198 (+ 4341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:49.835806 (+ 5608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:49.835815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:49.835895 (+ 80us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:49.836588 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:49.837189 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:49.837196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:49.837202 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:49.837483 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:49.838615 (+ 1132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:49.838624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:49.839050 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 02:38:49.842797 (+ 3747us) client_negotiation.cc:240] Negotiation successful
0620 02:38:49.843445 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23572,"thread_start_us":26247,"threads_started":1}
I20260620 02:38:49.849624 13534 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33285
I20260620 02:38:49.850063 13534 heartbeater.cc:464] Registering TS with master...
I20260620 02:38:49.851058 13534 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
W20260620 02:38:49.854308 13534 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33285 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:38:49.854590 13534 heartbeater.cc:464] Registering TS with master...
I20260620 02:38:49.854913 13534 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
I20260620 02:38:49.856213 13534 heartbeater.cc:464] Registering TS with master...
I20260620 02:38:49.856518 13534 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
W20260620 02:38:49.857761 13534 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:38:50.239391 13538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:38:50.239745 13538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:38:50.239959 13538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:38:50.250753 13538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:38:50.250926 13538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:38:50.251180 13538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:38:50.265739 13538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46049/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35169
--tserver_master_addrs=127.0.97.62:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:38:50.267311 13538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:38:50.268983 13538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:38:50.304795 13553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:50.306437 13555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:50.315363 13552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:50.316751 13538 server_base.cc:1034] running on GCE node
I20260620 02:38:50.317896 13538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:38:50.319276 13538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:38:50.320529 13538 hybrid_clock.cc:648] HybridClock initialized: now 1781923130320456 us; error 78 us; skew 500 ppm
I20260620 02:38:50.320933 13538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:38:50 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:50.324812 13538 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:38:50.338408 13538 webserver.cc:460] Webserver started at http://127.0.97.2:35169/ using document root <none> and password file <none>
I20260620 02:38:50.339558 13538 fs_manager.cc:359] Metadata directory not provided
I20260620 02:38:50.339895 13538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:38:50 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:38:50.394402 13538 fs_manager.cc:709] Time spent opening directory manager: real 0.023s user 0.002s sys 0.019s
I20260620 02:38:50.408560 13565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:50.410828 13538 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.001s sys 0.011s
I20260620 02:38:50.411146 13538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "5a2f81bb775d42b48400dd015dfd8802"
format_stamp: "Formatted at 2026-06-20 02:38:46 on dist-test-slave-rpg6"
server_key: "61ba3b59728669e7aa1f5536b74ced30"
server_key_iv: "c91e18903455a88ecd4b5850477b81b5"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:50.412799 13538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 02:38:50 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781923130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:50.492507 13538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:38:50.498389 13538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:38:50.498726 13538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:38:50.499917 13538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:38:50.501698 13538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:38:50.501827 13538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:50.501976 13538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:38:50.502066 13538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:50.617522 13538 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34903
I20260620 02:38:50.617692 13678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34903 every 8 connection(s)
I20260620 02:38:50.619724 13538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:38:50.623790 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 13538
I20260620 02:38:50.624883 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46049/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34785
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33285
--webserver_port=46205
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46049/kms with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:38:50 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:38:50.663650 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:50.623457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33285 (local address 127.0.97.2:56641)
0620 02:38:50.633160 (+ 9703us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:50.633257 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:50.637691 (+ 4434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:50.641039 (+ 3348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:50.641055 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:50.641760 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:50.642755 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:50.642793 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:50.644698 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:50.644705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:50.647074 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:50.647113 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:50.648655 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:50.648663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:50.649014 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:50.649942 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:50.649978 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:50.653317 (+ 3339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:50.658046 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:50.658058 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:50.658082 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:50.658825 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:50.659383 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:50.659389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:50.659392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:50.659701 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:50.660553 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:50.660563 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:50.661082 (+ 519us) client_negotiation.cc:769] Sending connection context
0620 02:38:50.661448 (+ 366us) client_negotiation.cc:240] Negotiation successful
0620 02:38:50.662112 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9214,"thread_start_us":12283,"threads_started":1}
I20260620 02:38:50.665092 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:50.624103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56641 (local address 127.0.97.62:33285)
0620 02:38:50.640244 (+ 16141us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:50.640250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:50.640281 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:38:50.640360 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:50.640366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:50.640467 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:50.643034 (+ 2567us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:50.643246 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:50.644438 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:50.647401 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:50.648104 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:50.648265 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:50.653701 (+ 5436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:50.653709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:50.653725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:50.653775 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:50.657774 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:50.659031 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:50.659038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:50.659042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:50.659221 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:50.659905 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:50.659911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:50.659914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:50.660314 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:50.660461 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:50.664376 (+ 3915us) server_negotiation.cc:299] Negotiation successful
0620 02:38:50.664684 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15975,"thread_start_us":19725,"threads_started":1}
I20260620 02:38:50.667914 13679 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33285
I20260620 02:38:50.668298 13679 heartbeater.cc:464] Registering TS with master...
I20260620 02:38:50.669238 13679 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
I20260620 02:38:50.671705 13337 ts_manager.cc:194] Registered new tserver with Master: 5a2f81bb775d42b48400dd015dfd8802 (127.0.97.2:34903)
I20260620 02:38:50.674386 13337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56641
I20260620 02:38:50.858019 13534 heartbeater.cc:464] Registering TS with master...
I20260620 02:38:50.858533 13534 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
I20260620 02:38:50.860214 13337 ts_manager.cc:194] Registered new tserver with Master: 02112b4f3b654539bef7497035387aa7 (127.0.97.1:36999)
I20260620 02:38:50.861553 13337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38855
Jun 20 02:38:51 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 02:38:51.187472 13683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:38:51.188136 13683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:38:51.188370 13683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:38:51.202590 13683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:38:51.202920 13683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:38:51.203186 13683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:38:51.222337 13683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46049/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46205
--tserver_master_addrs=127.0.97.62:33285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:38:51.224682 13683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:38:51.226917 13683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:38:51.256400 13688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:51.265893 13689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:51.266355 13683 server_base.cc:1034] running on GCE node
W20260620 02:38:51.266788 13692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:51.267799 13683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:38:51.269366 13683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:38:51.270758 13683 hybrid_clock.cc:648] HybridClock initialized: now 1781923131270684 us; error 64 us; skew 500 ppm
I20260620 02:38:51.271178 13683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:38:51 dist-test-slave-rpg6 krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:51.275867 13683 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:38:51.278048 13683 webserver.cc:460] Webserver started at http://127.0.97.3:46205/ using document root <none> and password file <none>
I20260620 02:38:51.279057 13683 fs_manager.cc:359] Metadata directory not provided
I20260620 02:38:51.279232 13683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:38:51 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:38:51.327580 13683 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 02:38:51.339164 13699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:51.341780 13683 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.007s sys 0.004s
I20260620 02:38:51.342098 13683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "70d920ad3c9a41f09d48dd9ae0218f17"
format_stamp: "Formatted at 2026-06-20 02:38:47 on dist-test-slave-rpg6"
server_key: "92904083b3f11fc746afdfcc0cc8075e"
server_key_iv: "a4d2a6d2d147dd9ed47c55278761d535"
server_key_version: "kuduclusterkey@0"
I20260620 02:38:51.343777 13683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:38:51.426918 13683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:38:51.432776 13683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:38:51.433154 13683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:38:51.434378 13683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:38:51.436228 13683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:38:51.436363 13683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:51.436520 13683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:38:51.436616 13683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:38:51 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:38:51.680174 13679 heartbeater.cc:503] Master 127.0.97.62:33285 was elected leader, sending a full tablet report...
I20260620 02:38:51.733140 13683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34785
I20260620 02:38:51.733160 13812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34785 every 8 connection(s)
I20260620 02:38:51.735003 13683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:38:51.744328 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 13683
Jun 20 02:38:51 dist-test-slave-rpg6 krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:38:51.771081 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:51.739211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33285 (local address 127.0.97.3:46425)
0620 02:38:51.740514 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:51.740602 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:51.743404 (+ 2802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:51.749393 (+ 5989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:51.749410 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:51.750136 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:51.751136 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:51.751168 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.752938 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.752946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:51.755146 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:51.755166 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.756556 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.756564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:51.756882 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:51.757798 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:51.757832 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:51.760981 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:51.765710 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:51.765722 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:51.765744 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:51.766443 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:51.766974 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:51.766980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:51.766984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:51.767277 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:51.768113 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:51.768123 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:51.768613 (+ 490us) client_negotiation.cc:769] Sending connection context
0620 02:38:51.769003 (+ 390us) client_negotiation.cc:240] Negotiation successful
0620 02:38:51.769625 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":673,"threads_started":1}
I20260620 02:38:51.772327 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:51.739861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46425 (local address 127.0.97.62:33285)
0620 02:38:51.748321 (+ 8460us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:51.748330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:51.748362 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:38:51.748472 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:51.748478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:51.748581 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:51.748760 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:51.751424 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.752705 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.755369 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.756077 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.756207 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:51.761320 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:51.761328 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:51.761332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:51.761396 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:51.765442 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:51.766623 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:51.766629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:51.766634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:51.766819 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:51.767428 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:51.767433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:51.767436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:51.767889 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:51.768025 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:51.771635 (+ 3610us) server_negotiation.cc:299] Negotiation successful
0620 02:38:51.771940 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8276,"thread_start_us":8150,"threads_started":1}
I20260620 02:38:51.775038 13813 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33285
I20260620 02:38:51.775460 13813 heartbeater.cc:464] Registering TS with master...
I20260620 02:38:51.776835 13813 heartbeater.cc:511] Master 127.0.97.62:33285 requested a full tablet report, sending...
I20260620 02:38:51.778847 13337 ts_manager.cc:194] Registered new tserver with Master: 70d920ad3c9a41f09d48dd9ae0218f17 (127.0.97.3:34785)
I20260620 02:38:51.780117 13337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46425
I20260620 02:38:51.781903 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:51.745501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.0.97.62:33285)
0620 02:38:51.748554 (+ 3053us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:51.748560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:51.749152 (+ 592us) server_negotiation.cc:407] Connection header received
0620 02:38:51.749330 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:51.749335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:51.749415 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:51.749528 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:51.753068 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.754504 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.762604 (+ 8100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.763411 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.763567 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:51.773917 (+ 10350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:51.773923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:51.773927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:51.773970 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:51.777606 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:51.779365 (+ 1759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:51.779371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:51.779375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:51.779571 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:51.780054 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:51.780059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:51.780063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:51.780452 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:51.780756 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:51.781247 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 02:38:51.781506 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2953,"thread_start_us":2777,"threads_started":1}
I20260620 02:38:51.784977 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:38:51.810817 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:51.791551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.0.97.62:33285)
0620 02:38:51.791769 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:51.791776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:51.791802 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:38:51.792093 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:51.792101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:51.792202 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:51.792283 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:51.794646 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.796128 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.798369 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.799083 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.799223 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:51.803375 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:51.803382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:51.803387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:51.803443 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:51.807113 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:51.808146 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:51.808153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:51.808156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:51.808337 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:51.808866 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:51.808872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:51.808875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:51.809255 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:51.809502 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:51.810161 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 02:38:51.810428 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:38:51.818931 13337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:38:51.830963 13337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:38:51.865276 13534 heartbeater.cc:503] Master 127.0.97.62:33285 was elected leader, sending a full tablet report...
I20260620 02:38:51.896695 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:51.856626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.0.97.1:36999)
0620 02:38:51.873651 (+ 17025us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:51.873662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:51.878088 (+ 4426us) server_negotiation.cc:407] Connection header received
0620 02:38:51.878535 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:51.878546 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:51.878853 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:51.879018 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:38:51.881811 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.883444 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.889384 (+ 5940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.891665 (+ 2281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.891902 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:51.892277 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:51.896088 (+ 3811us) server_negotiation.cc:299] Negotiation successful
0620 02:38:51.896327 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16793,"thread_start_us":16618,"threads_started":1}
I20260620 02:38:51.897178 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:51.856010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36999 (local address 127.0.0.1:37594)
0620 02:38:51.877905 (+ 21895us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:51.877958 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:51.878260 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:51.879492 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:51.879501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:51.879552 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:38:51.881587 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:51.881600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.883668 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.883673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:51.887183 (+ 3510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:51.887217 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.892530 (+ 5313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.892542 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:51.892907 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:51.892986 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 02:38:51.893181 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 02:38:51.893492 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21269,"mutex_wait_us":1791,"spinlock_wait_cycles":22016,"thread_start_us":21063,"threads_started":1}
I20260620 02:38:51.908911 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:51.884014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34903 (local address 127.0.0.1:36498)
0620 02:38:51.897629 (+ 13615us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:51.897651 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:51.897821 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:51.899014 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:51.899020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:51.899050 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:38:51.899250 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:51.899262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.903274 (+ 4012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.903286 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:51.905653 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:51.905676 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.908047 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.908054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:51.908250 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:51.908289 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:38:51.908395 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 02:38:51.908591 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13492,"thread_start_us":21864,"threads_started":1}
I20260620 02:38:51.909765 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:51.880295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.0.97.2:34903)
0620 02:38:51.888414 (+ 8119us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:51.888424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:51.898000 (+ 9576us) server_negotiation.cc:407] Connection header received
0620 02:38:51.898133 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:51.898142 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:51.898470 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:51.898733 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:38:51.899452 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.901482 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.905910 (+ 4428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.907473 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.907644 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:51.907931 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:51.909233 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0620 02:38:51.909427 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7914,"thread_start_us":7713,"threads_started":1}
I20260620 02:38:51.910907 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:51.856717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.0.97.3:34785)
0620 02:38:51.871860 (+ 15143us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:51.871871 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:51.878488 (+ 6617us) server_negotiation.cc:407] Connection header received
0620 02:38:51.878620 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:51.878633 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:51.878893 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:51.879078 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:38:51.880381 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.883113 (+ 2732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.906970 (+ 23857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.908630 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.908745 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:51.908986 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:51.910368 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0620 02:38:51.910563 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14893,"thread_start_us":15165,"threads_started":1}
I20260620 02:38:51.910928 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:51.856323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34785 (local address 127.0.0.1:35330)
0620 02:38:51.878008 (+ 21685us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:51.878030 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:51.878259 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:51.879494 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:51.879502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:51.879546 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:38:51.880121 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:51.880138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.883367 (+ 3229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.883376 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:51.906721 (+ 23345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:51.906762 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:51.909451 (+ 2689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:51.909458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:51.909711 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:51.909778 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 02:38:51.910258 (+ 480us) client_negotiation.cc:240] Negotiation successful
0620 02:38:51.910414 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21540,"mutex_wait_us":17368,"spinlock_wait_cycles":9216,"thread_start_us":17541,"threads_started":1}
I20260620 02:38:51.910804 13469 tablet_service.cc:1467] Processing CreateTablet for tablet 2d567fe7995b4d5a98363110749c774e (DEFAULT_TABLE table=test-table [id=bf49b5d42ea44e7fb6fe4a5135efcb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:38:51.912575 13469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d567fe7995b4d5a98363110749c774e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:38:51.915248 13747 tablet_service.cc:1467] Processing CreateTablet for tablet 2d567fe7995b4d5a98363110749c774e (DEFAULT_TABLE table=test-table [id=bf49b5d42ea44e7fb6fe4a5135efcb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:38:51.918411 13747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d567fe7995b4d5a98363110749c774e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:38:51.920660 13614 tablet_service.cc:1467] Processing CreateTablet for tablet 2d567fe7995b4d5a98363110749c774e (DEFAULT_TABLE table=test-table [id=bf49b5d42ea44e7fb6fe4a5135efcb67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:38:51.922317 13614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d567fe7995b4d5a98363110749c774e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:38:51.935488 13841 tablet_bootstrap.cc:492] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: Bootstrap starting.
I20260620 02:38:51.942133 13841 tablet_bootstrap.cc:654] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: Neither blocks nor log segments found. Creating new log.
I20260620 02:38:51.944159 13841 log.cc:824] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: Log is configured to *not* fsync() on all Append() calls
I20260620 02:38:51.946156 13842 tablet_bootstrap.cc:492] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: Bootstrap starting.
I20260620 02:38:51.947978 13843 tablet_bootstrap.cc:492] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802: Bootstrap starting.
I20260620 02:38:51.952061 13841 tablet_bootstrap.cc:492] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: No bootstrap required, opened a new log
I20260620 02:38:51.952395 13842 tablet_bootstrap.cc:654] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: Neither blocks nor log segments found. Creating new log.
I20260620 02:38:51.952517 13841 ts_tablet_manager.cc:1397] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260620 02:38:51.952665 13843 tablet_bootstrap.cc:654] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802: Neither blocks nor log segments found. Creating new log.
I20260620 02:38:51.954021 13842 log.cc:824] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: Log is configured to *not* fsync() on all Append() calls
I20260620 02:38:51.954088 13843 log.cc:824] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802: Log is configured to *not* fsync() on all Append() calls
I20260620 02:38:51.963973 13843 tablet_bootstrap.cc:492] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802: No bootstrap required, opened a new log
I20260620 02:38:51.964408 13843 ts_tablet_manager.cc:1397] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260620 02:38:51.965624 13842 tablet_bootstrap.cc:492] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: No bootstrap required, opened a new log
I20260620 02:38:51.966029 13842 ts_tablet_manager.cc:1397] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.009s
I20260620 02:38:51.966615 13841 raft_consensus.cc:348] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02112b4f3b654539bef7497035387aa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36999 } } peers { permanent_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34785 } } peers { permanent_uuid: "5a2f81bb775d42b48400dd015dfd8802" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260620 02:38:51.967236 13841 raft_consensus.cc:374] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:38:51.967408 13841 raft_consensus.cc:729] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02112b4f3b654539bef7497035387aa7, State: Initialized, Role: FOLLOWER
I20260620 02:38:51.968387 13841 consensus_queue.cc:260] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02112b4f3b654539bef7497035387aa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36999 } } peers { permanent_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34785 } } peers { permanent_uuid: "5a2f81bb775d42b48400dd015dfd8802" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260620 02:38:51.980137 13842 raft_consensus.cc:348] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02112b4f3b654539bef7497035387aa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36999 } } peers { permanent_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34785 } } peers { permanent_uuid: "5a2f81bb775d42b48400dd015dfd8802" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260620 02:38:51.980806 13842 raft_consensus.cc:374] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:38:51.980973 13842 raft_consensus.cc:729] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d920ad3c9a41f09d48dd9ae0218f17, State: Initialized, Role: FOLLOWER
I20260620 02:38:51.981482 13843 raft_consensus.cc:348] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02112b4f3b654539bef7497035387aa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36999 } } peers { permanent_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34785 } } peers { permanent_uuid: "5a2f81bb775d42b48400dd015dfd8802" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260620 02:38:51.981882 13842 consensus_queue.cc:260] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02112b4f3b654539bef7497035387aa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36999 } } peers { permanent_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34785 } } peers { permanent_uuid: "5a2f81bb775d42b48400dd015dfd8802" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260620 02:38:51.983732 13841 ts_tablet_manager.cc:1428] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: Time spent starting tablet: real 0.031s user 0.020s sys 0.009s
I20260620 02:38:51.984264 13843 raft_consensus.cc:374] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:38:51.984436 13843 raft_consensus.cc:729] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2f81bb775d42b48400dd015dfd8802, State: Initialized, Role: FOLLOWER
I20260620 02:38:51.985244 13843 consensus_queue.cc:260] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02112b4f3b654539bef7497035387aa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36999 } } peers { permanent_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34785 } } peers { permanent_uuid: "5a2f81bb775d42b48400dd015dfd8802" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260620 02:38:51.990564 13842 ts_tablet_manager.cc:1428] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: Time spent starting tablet: real 0.024s user 0.014s sys 0.005s
I20260620 02:38:51.991370 13813 heartbeater.cc:503] Master 127.0.97.62:33285 was elected leader, sending a full tablet report...
I20260620 02:38:51.992909 13843 ts_tablet_manager.cc:1428] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802: Time spent starting tablet: real 0.028s user 0.019s sys 0.006s
W20260620 02:38:51.993345 13814 tablet.cc:2369] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:38:52.020722 13849 raft_consensus.cc:482] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:38:52.021039 13849 raft_consensus.cc:504] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02112b4f3b654539bef7497035387aa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36999 } } peers { permanent_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34785 } } peers { permanent_uuid: "5a2f81bb775d42b48400dd015dfd8802" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260620 02:38:52.023929 13849 leader_election.cc:288] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02112b4f3b654539bef7497035387aa7 (127.0.97.1:36999), 70d920ad3c9a41f09d48dd9ae0218f17 (127.0.97.3:34785)
I20260620 02:38:52.032404 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:52.024859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52945 (local address 127.0.97.3:34785)
0620 02:38:52.025063 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:52.025069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:52.025758 (+ 689us) server_negotiation.cc:407] Connection header received
0620 02:38:52.026099 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:52.026105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:52.026192 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:52.026321 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:38:52.026990 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.028049 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.030063 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.031106 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.031236 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:52.031352 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:52.031957 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 02:38:52.032121 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:38:52.033082 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:52.024697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42035 (local address 127.0.97.1:36999)
0620 02:38:52.024976 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:52.024985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:52.026087 (+ 1102us) server_negotiation.cc:407] Connection header received
0620 02:38:52.026513 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:52.026520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:52.026617 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:52.026685 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:38:52.027409 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.028286 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.030700 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.032002 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.032133 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:52.032238 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:52.032590 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 02:38:52.032728 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:38:52.033465 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:52.024440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34785 (local address 127.0.97.2:52945)
0620 02:38:52.025587 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:52.025613 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:52.025905 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:52.026373 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:52.026381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:52.026427 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:38:52.026820 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:52.026836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.028195 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.028201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:52.029895 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:52.029908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.031500 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.031506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:52.031695 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:52.031752 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 02:38:52.032901 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0620 02:38:52.033070 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"thread_start_us":1223,"threads_started":1}
I20260620 02:38:52.033701 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:52.024541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36999 (local address 127.0.97.2:42035)
0620 02:38:52.025992 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:52.026009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:52.026306 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:52.026917 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:52.026922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:52.026949 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:38:52.027247 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:52.027256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.028455 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.028460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:52.030516 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:52.030531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.032226 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.032232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:52.032391 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:52.032424 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:38:52.033142 (+ 718us) client_negotiation.cc:240] Negotiation successful
0620 02:38:52.033307 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"thread_start_us":684,"threads_started":1}
I20260620 02:38:52.034648 13768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d567fe7995b4d5a98363110749c774e" candidate_uuid: "5a2f81bb775d42b48400dd015dfd8802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" is_pre_election: true
I20260620 02:38:52.034879 13489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d567fe7995b4d5a98363110749c774e" candidate_uuid: "5a2f81bb775d42b48400dd015dfd8802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02112b4f3b654539bef7497035387aa7" is_pre_election: true
I20260620 02:38:52.035256 13768 raft_consensus.cc:2454] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2f81bb775d42b48400dd015dfd8802 in term 0.
I20260620 02:38:52.035444 13489 raft_consensus.cc:2454] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a2f81bb775d42b48400dd015dfd8802 in term 0.
I20260620 02:38:52.036136 13567 leader_election.cc:302] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02112b4f3b654539bef7497035387aa7, 5a2f81bb775d42b48400dd015dfd8802; no voters:
I20260620 02:38:52.036610 13849 raft_consensus.cc:2781] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:38:52.036773 13849 raft_consensus.cc:482] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:38:52.036882 13849 raft_consensus.cc:3037] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:38:52.039245 13849 raft_consensus.cc:504] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02112b4f3b654539bef7497035387aa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36999 } } peers { permanent_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34785 } } peers { permanent_uuid: "5a2f81bb775d42b48400dd015dfd8802" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260620 02:38:52.040153 13849 leader_election.cc:288] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [CANDIDATE]: Term 1 election: Requested vote from peers 02112b4f3b654539bef7497035387aa7 (127.0.97.1:36999), 70d920ad3c9a41f09d48dd9ae0218f17 (127.0.97.3:34785)
I20260620 02:38:52.040812 13768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d567fe7995b4d5a98363110749c774e" candidate_uuid: "5a2f81bb775d42b48400dd015dfd8802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d920ad3c9a41f09d48dd9ae0218f17"
I20260620 02:38:52.040804 13489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d567fe7995b4d5a98363110749c774e" candidate_uuid: "5a2f81bb775d42b48400dd015dfd8802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02112b4f3b654539bef7497035387aa7"
I20260620 02:38:52.041059 13489 raft_consensus.cc:3037] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:38:52.041060 13768 raft_consensus.cc:3037] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:38:52.044709 13489 raft_consensus.cc:2454] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2f81bb775d42b48400dd015dfd8802 in term 1.
I20260620 02:38:52.044709 13768 raft_consensus.cc:2454] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2f81bb775d42b48400dd015dfd8802 in term 1.
I20260620 02:38:52.045288 13567 leader_election.cc:302] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02112b4f3b654539bef7497035387aa7, 5a2f81bb775d42b48400dd015dfd8802; no voters:
I20260620 02:38:52.046278 13849 raft_consensus.cc:2781] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:38:52.047330 13849 raft_consensus.cc:686] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 1 LEADER]: Becoming Leader. State: Replica: 5a2f81bb775d42b48400dd015dfd8802, State: Running, Role: LEADER
I20260620 02:38:52.048169 13849 consensus_queue.cc:237] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02112b4f3b654539bef7497035387aa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36999 } } peers { permanent_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34785 } } peers { permanent_uuid: "5a2f81bb775d42b48400dd015dfd8802" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
W20260620 02:38:52.053938 13535 tablet.cc:2369] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:38:52.053938 13336 catalog_manager.cc:5495] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a2f81bb775d42b48400dd015dfd8802 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5a2f81bb775d42b48400dd015dfd8802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02112b4f3b654539bef7497035387aa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a2f81bb775d42b48400dd015dfd8802" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } health_report { overall_health: HEALTHY } } }
I20260620 02:38:52.088757 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:52.080359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.0.97.2:34903)
0620 02:38:52.080551 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:52.080556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:52.080754 (+ 198us) server_negotiation.cc:407] Connection header received
0620 02:38:52.080928 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:52.080934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:52.081107 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:52.081400 (+ 293us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:38:52.081891 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.083022 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.085257 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.085854 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.085969 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:52.086785 (+ 816us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:38:52.087760 (+ 975us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:38:52.088252 (+ 492us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:52.088312 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 02:38:52.088399 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:38:52.097955 13489 raft_consensus.cc:1261] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7 [term 1 FOLLOWER]: Refusing update from remote peer 5a2f81bb775d42b48400dd015dfd8802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:38:52.097955 13768 raft_consensus.cc:1261] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17 [term 1 FOLLOWER]: Refusing update from remote peer 5a2f81bb775d42b48400dd015dfd8802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:38:52.099936 13852 consensus_queue.cc:1035] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02112b4f3b654539bef7497035387aa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:38:52.100445 13849 consensus_queue.cc:1035] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70d920ad3c9a41f09d48dd9ae0218f17" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:38:52.115640 13855 mvcc.cc:205] Tried to move back new op lower bound from 7298757149061074944 to 7298757148874141696. Current Snapshot: MvccSnapshot[applied={T|T < 7298757149061074944}]
I20260620 02:38:52.117731 13856 mvcc.cc:205] Tried to move back new op lower bound from 7298757149061074944 to 7298757148874141696. Current Snapshot: MvccSnapshot[applied={T|T < 7298757149061074944}]
I20260620 02:38:52.151021 13336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:38:52.151347 13336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:38:52.157037 13336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c78433e90108484693583312f062934a: Sending DeleteTablet for 3 replicas of tablet 2d567fe7995b4d5a98363110749c774e
I20260620 02:38:52.158815 13747 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d567fe7995b4d5a98363110749c774e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:38:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35330
I20260620 02:38:52.159586 13469 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d567fe7995b4d5a98363110749c774e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:38:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37594
I20260620 02:38:52.161062 13746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d567fe7995b4d5a98363110749c774e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:38:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35330
I20260620 02:38:52.162195 13614 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d567fe7995b4d5a98363110749c774e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:38:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36498
I20260620 02:38:52.169060 13866 tablet_replica.cc:330] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802: stopping tablet replica
I20260620 02:38:52.170533 13866 raft_consensus.cc:2229] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:38:52.171288 13866 raft_consensus.cc:2258] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:38:52.173094 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 13374
I20260620 02:38:52.173449 13864 tablet_replica.cc:330] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: stopping tablet replica
I20260620 02:38:52.175063 13865 tablet_replica.cc:330] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: stopping tablet replica
I20260620 02:38:52.175503 13865 raft_consensus.cc:2229] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:38:52.176136 13865 raft_consensus.cc:2258] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:38:52.173954 13864 raft_consensus.cc:2229] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:38:52.177122 13864 raft_consensus.cc:2258] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:38:52.179585 13864 ts_tablet_manager.cc:1905] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:38:52.187052 13865 ts_tablet_manager.cc:1905] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:38:52.194495 13865 ts_tablet_manager.cc:1918] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:38:52.196787 13864 ts_tablet_manager.cc:1918] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:38:52.197072 13864 log.cc:1196] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/2d567fe7995b4d5a98363110749c774e
I20260620 02:38:52.197301 13865 log.cc:1196] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/2d567fe7995b4d5a98363110749c774e
I20260620 02:38:52.198058 13864 ts_tablet_manager.cc:1939] T 2d567fe7995b4d5a98363110749c774e P 70d920ad3c9a41f09d48dd9ae0218f17: Deleting consensus metadata
I20260620 02:38:52.198732 13865 ts_tablet_manager.cc:1939] T 2d567fe7995b4d5a98363110749c774e P 02112b4f3b654539bef7497035387aa7: Deleting consensus metadata
I20260620 02:38:52.208115 13322 catalog_manager.cc:4841] TS 02112b4f3b654539bef7497035387aa7 (127.0.97.1:36999): tablet 2d567fe7995b4d5a98363110749c774e (table test-table [id=bf49b5d42ea44e7fb6fe4a5135efcb67]) successfully deleted
I20260620 02:38:52.208648 13323 catalog_manager.cc:4841] TS 70d920ad3c9a41f09d48dd9ae0218f17 (127.0.97.3:34785): tablet 2d567fe7995b4d5a98363110749c774e (table test-table [id=bf49b5d42ea44e7fb6fe4a5135efcb67]) successfully deleted
W20260620 02:38:52.209121 13323 catalog_manager.cc:4808] TS 70d920ad3c9a41f09d48dd9ae0218f17 (127.0.97.3:34785): delete failed for tablet 2d567fe7995b4d5a98363110749c774e because the tablet was not found. No further retry: Not found: Tablet not found: 2d567fe7995b4d5a98363110749c774e
I20260620 02:38:52.209833 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:52.174217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.0.97.1:36999)
0620 02:38:52.174405 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:52.174412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:52.174438 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:38:52.174681 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:52.174688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:52.174806 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:52.174915 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:52.183507 (+ 8592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.184711 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.187006 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.187704 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.187877 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:52.193203 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:52.193220 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:52.193243 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:52.193300 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:52.204138 (+ 10838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:52.205930 (+ 1792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:52.205942 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:52.205948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:52.206240 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:52.207010 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:52.207015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:52.207019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:52.207745 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:52.208147 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:52.209160 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0620 02:38:52.209408 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:38:52.212628 13866 ts_tablet_manager.cc:1905] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:38:52.212996 13529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:38:52.220610 13866 ts_tablet_manager.cc:1918] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:38:52.220913 13866 log.cc:1196] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/2d567fe7995b4d5a98363110749c774e
I20260620 02:38:52.221570 13866 ts_tablet_manager.cc:1939] T 2d567fe7995b4d5a98363110749c774e P 5a2f81bb775d42b48400dd015dfd8802: Deleting consensus metadata
I20260620 02:38:52.223738 13322 catalog_manager.cc:4841] TS 5a2f81bb775d42b48400dd015dfd8802 (127.0.97.2:34903): tablet 2d567fe7995b4d5a98363110749c774e (table test-table [id=bf49b5d42ea44e7fb6fe4a5135efcb67]) successfully deleted
I20260620 02:38:52.365904 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 13374
I20260620 02:38:52.389729 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 13538
I20260620 02:38:52.405608 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:52.390662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.0.97.2:34903)
0620 02:38:52.390957 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:52.390968 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:52.391003 (+ 35us) server_negotiation.cc:407] Connection header received
0620 02:38:52.391084 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:52.391088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:52.391153 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:52.391300 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:52.392734 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.393598 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.395040 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.395502 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.395654 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:52.398555 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:52.398573 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:52.398590 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:52.398633 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:52.401861 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:52.402786 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:52.402794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:52.402797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:52.402951 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:52.403412 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:52.403420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:52.403426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:52.404215 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:52.404572 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:52.404969 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 02:38:52.405241 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260620 02:38:52.406433 13674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:38:52.526146 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 13538
I20260620 02:38:52.545149 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 13683
I20260620 02:38:52.560720 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:52.546072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.0.97.3:34785)
0620 02:38:52.546550 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:52.546555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:52.546572 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:38:52.546627 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:52.546630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:52.546714 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:52.546824 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:52.548371 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.549062 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.550485 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:52.550966 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:52.551064 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:52.554256 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:52.554274 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:52.554291 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:52.554331 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:52.557286 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:52.558148 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:52.558156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:52.558158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:52.558312 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:52.558730 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:52.558733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:52.558735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:52.559234 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:52.559470 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:52.560239 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 02:38:52.560430 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":269,"threads_started":1}
I20260620 02:38:52.561501 13808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:38:52.655135 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 13683
I20260620 02:38:52.672847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 13304
I20260620 02:38:52.674103 13367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:38:52.790401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 13304
I20260620 02:38:52.814301 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 02:38:53.151486 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 02:38:53.151968 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 02:38:53.499258 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 02:38:53.499 UTC [12356] LOG: received smart shutdown request
2026-06-20 02:38:53.503 UTC [12356] LOG: background worker "logical replication launcher" (PID 12365) exited with exit code 1
2026-06-20 02:38:53.503 UTC [12360] LOG: shutting down
2026-06-20 02:38:53.724 UTC [12356] LOG: database system is shut down
2026-06-20T02:38:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58102 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:38:53 dist-test-slave-rpg6 krb5kdc[13880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:38:53 dist-test-slave-rpg6 krb5kdc[13880](info): set up 2 sockets
Jun 20 02:38:53 dist-test-slave-rpg6 krb5kdc[13880](info): commencing operation
krb5kdc: starting...
W20260620 02:38:55.012799 388 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:38:55 dist-test-slave-rpg6 krb5kdc[13880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:38:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:38:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:38:55.264089 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36073
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36073,127.0.97.61:33345,127.0.97.60:34223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:38:55.616552 13893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:38:55.616942 13893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:38:55.617122 13893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:38:55.624933 13893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:38:55.625139 13893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:38:55.625252 13893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:38:55.625360 13893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:38:55.625454 13893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:38:55.635916 13893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36073,127.0.97.61:33345,127.0.97.60:34223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:38:55.637440 13893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:38:55.639071 13893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:38:55.652392 13898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:55.652420 13899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:55.652350 13901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:55.653295 13893 server_base.cc:1034] running on GCE node
I20260620 02:38:55.654177 13893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:38:55.655762 13893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:38:55.657069 13893 hybrid_clock.cc:648] HybridClock initialized: now 1781923135657022 us; error 39 us; skew 500 ppm
I20260620 02:38:55.657574 13893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:38:55 dist-test-slave-rpg6 krb5kdc[13880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:55.673911 13893 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:38:55.675659 13893 webserver.cc:460] Webserver started at http://127.0.97.62:36869/ using document root <none> and password file <none>
I20260620 02:38:55.676537 13893 fs_manager.cc:359] Metadata directory not provided
I20260620 02:38:55.676707 13893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:38:55.677040 13893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:38:55.680596 13893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "5e41de5bb3b64703ad470af1d519368c"
format_stamp: "Formatted at 2026-06-20 02:38:55 on dist-test-slave-rpg6"
server_key: "94c040f6fc7819214f6a23dd69c30cf6"
server_key_iv: "9a1afc3497c9f554e17b09ff94c570b3"
server_key_version: "encryptionkey@0"
I20260620 02:38:55.681516 13893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "5e41de5bb3b64703ad470af1d519368c"
format_stamp: "Formatted at 2026-06-20 02:38:55 on dist-test-slave-rpg6"
server_key: "94c040f6fc7819214f6a23dd69c30cf6"
server_key_iv: "9a1afc3497c9f554e17b09ff94c570b3"
server_key_version: "encryptionkey@0"
I20260620 02:38:55.687978 13893 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:38:55.692271 13908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:55.694031 13893 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:38:55.694267 13893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "5e41de5bb3b64703ad470af1d519368c"
format_stamp: "Formatted at 2026-06-20 02:38:55 on dist-test-slave-rpg6"
server_key: "94c040f6fc7819214f6a23dd69c30cf6"
server_key_iv: "9a1afc3497c9f554e17b09ff94c570b3"
server_key_version: "encryptionkey@0"
I20260620 02:38:55.694510 13893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:38:55.714308 13893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:38:55.719425 13893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:38:55.719784 13893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:38:55.741930 13893 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36073
I20260620 02:38:55.741971 13959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36073 every 8 connection(s)
I20260620 02:38:55.743455 13893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:38:55.747625 13960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:38:55.749747 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 13893
I20260620 02:38:55.749997 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:38:55.750865 388 external_mini_cluster.cc:1428] Setting key beea6adcd652330b654009f743e926dc
I20260620 02:38:55.758697 13960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 20 02:38:55 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:38:55.788743 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:55.763139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36073 (local address 127.0.0.1:44810)
0620 02:38:55.764766 (+ 1627us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:55.764813 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:55.766874 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:55.768138 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:55.768145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:55.768977 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:55.769426 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:55.769446 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:55.771294 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:55.771302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:55.773494 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:55.773519 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:55.774652 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:55.774670 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:55.774896 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:55.775577 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:55.775619 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:55.779670 (+ 4051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:55.783506 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:55.783526 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:55.783542 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:55.784294 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:55.784898 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:55.784902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:55.784904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:55.785074 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:55.786301 (+ 1227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:55.786309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:55.786771 (+ 462us) client_negotiation.cc:769] Sending connection context
0620 02:38:55.787034 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 02:38:55.787507 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"thread_start_us":844,"threads_started":1}
I20260620 02:38:55.788800 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:55.763147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.0.97.62:36073)
0620 02:38:55.764152 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:55.764168 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:55.764972 (+ 804us) server_negotiation.cc:407] Connection header received
0620 02:38:55.767356 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:55.767363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:55.767786 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:55.768106 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:55.769657 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:55.771060 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:55.773713 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:55.774340 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:55.774540 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:55.780053 (+ 5513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:55.780065 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:55.780080 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:55.780126 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:55.783300 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:55.784484 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:55.784492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:55.784495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:55.784756 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:55.785216 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:55.785223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:55.785228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:55.786057 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:55.786384 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:55.787252 (+ 868us) server_negotiation.cc:299] Negotiation successful
0620 02:38:55.787738 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":729,"thread_start_us":462,"threads_started":1}
I20260620 02:38:55.792449 13960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:38:55.794142 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:55.793555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54616)
0620 02:38:55.793799 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:55.793953 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 02:38:55.794567 13912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
I20260620 02:38:55.797261 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:55.796791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54628)
0620 02:38:55.797003 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:55.797088 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 02:38:55.797585 13960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33345: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 02:38:55.812398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33345
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36073,127.0.97.61:33345,127.0.97.60:34223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:38:55.846980 13960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } attempt: 1
I20260620 02:38:55.848523 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:55.847984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54640)
0620 02:38:55.848274 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:55.848369 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260620 02:38:55.849813 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:55.849401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54650)
0620 02:38:55.849564 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:55.849663 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:38:55.850023 13960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33345: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
I20260620 02:38:55.918298 13960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } attempt: 2
I20260620 02:38:55.919680 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:55.919174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54662)
0620 02:38:55.919442 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:55.919533 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 02:38:55.920774 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:55.920396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54672)
0620 02:38:55.920547 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:55.920639 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:38:55.920954 13960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33345: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
I20260620 02:38:56.012308 13960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } attempt: 3
I20260620 02:38:56.013672 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.013042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54686)
0620 02:38:56.013356 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.013487 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 02:38:56.016103 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.015447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54694)
0620 02:38:56.015706 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.015865 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 02:38:56.016446 13960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33345: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
I20260620 02:38:56.159773 13960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } attempt: 4
I20260620 02:38:56.161118 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.160562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54698)
0620 02:38:56.160875 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.160977 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 02:38:56.162323 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.161928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54704)
0620 02:38:56.162086 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.162192 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 02:38:56.162575 13960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33345: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33345: connect: Connection refused (error 111)
W20260620 02:38:56.170887 13967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:38:56.171298 13967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:38:56.171458 13967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:38:56.179265 13967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:38:56.179454 13967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:38:56.179579 13967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:38:56.179692 13967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:38:56.179781 13967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:38:56.190503 13967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36073,127.0.97.61:33345,127.0.97.60:34223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:38:56.192078 13967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:38:56.193537 13967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:38:56.207144 13973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:56.207762 13975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:56.208207 13972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:56.209044 13967 server_base.cc:1034] running on GCE node
I20260620 02:38:56.209988 13967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:38:56.211738 13967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:38:56.213204 13967 hybrid_clock.cc:648] HybridClock initialized: now 1781923136213147 us; error 39 us; skew 500 ppm
I20260620 02:38:56.213696 13967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:56.218005 13967 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 02:38:56.219887 13967 webserver.cc:460] Webserver started at http://127.0.97.61:42471/ using document root <none> and password file <none>
I20260620 02:38:56.220647 13967 fs_manager.cc:359] Metadata directory not provided
I20260620 02:38:56.220783 13967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:38:56.221104 13967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:38:56.224318 13967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/data/instance:
uuid: "7047a6d4fc3648a093fdc7b006a34010"
format_stamp: "Formatted at 2026-06-20 02:38:56 on dist-test-slave-rpg6"
server_key: "dc94b17d3c78ac32797ee209a8a59344"
server_key_iv: "c5ff59b266a90360a843b4c12e5da5e4"
server_key_version: "encryptionkey@0"
I20260620 02:38:56.225171 13967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/wal/instance:
uuid: "7047a6d4fc3648a093fdc7b006a34010"
format_stamp: "Formatted at 2026-06-20 02:38:56 on dist-test-slave-rpg6"
server_key: "dc94b17d3c78ac32797ee209a8a59344"
server_key_iv: "c5ff59b266a90360a843b4c12e5da5e4"
server_key_version: "encryptionkey@0"
I20260620 02:38:56.231133 13967 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260620 02:38:56.235021 13982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:56.236836 13967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:38:56.237059 13967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/wal
uuid: "7047a6d4fc3648a093fdc7b006a34010"
format_stamp: "Formatted at 2026-06-20 02:38:56 on dist-test-slave-rpg6"
server_key: "dc94b17d3c78ac32797ee209a8a59344"
server_key_iv: "c5ff59b266a90360a843b4c12e5da5e4"
server_key_version: "encryptionkey@0"
I20260620 02:38:56.237344 13967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:38:56.267508 13967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:38:56.272394 13967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:38:56.272662 13967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:38:56.295480 13967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33345
I20260620 02:38:56.295506 14033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33345 every 8 connection(s)
I20260620 02:38:56.297678 13967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:38:56.298765 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 13967
I20260620 02:38:56.299002 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-1/wal/instance
I20260620 02:38:56.299738 388 external_mini_cluster.cc:1428] Setting key f6be9b57165286185354c823828fb96e
I20260620 02:38:56.302512 14034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:38:56.310112 14034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:38:56.333294 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.313370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.0.97.62:36073)
0620 02:38:56.314026 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.314035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.315413 (+ 1378us) server_negotiation.cc:407] Connection header received
0620 02:38:56.317864 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.317870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.317976 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.318112 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.319890 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.320672 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.322383 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.322826 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.322969 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.326833 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.326839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.326843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.326884 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.329845 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.330729 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.330735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.330739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.330872 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.331336 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.331342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.331345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.331657 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.331828 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.332630 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.332881 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":370,"threads_started":1}
I20260620 02:38:56.334336 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.314131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36073 (local address 127.0.0.1:44818)
0620 02:38:56.315198 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.315246 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:56.317488 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:56.318405 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:56.318415 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:56.318997 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:56.319627 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.319645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.320864 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.320868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.322221 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.322235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.323083 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.323090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.323326 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.324119 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:56.324142 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:56.326526 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:56.330042 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.330053 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.330074 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.330578 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.330992 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.330998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.331001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.331216 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.331809 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:56.331818 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:56.332248 (+ 430us) client_negotiation.cc:769] Sending connection context
0620 02:38:56.332543 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 02:38:56.333125 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":454,"threads_started":1}
I20260620 02:38:56.336565 14034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:38:56.353257 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.337427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54712)
0620 02:38:56.337617 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.337645 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:56.337761 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:56.339241 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:56.339246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:56.339743 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:56.339970 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.339979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.341488 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.341494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.342471 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.342484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.343321 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.343326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.343564 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.344247 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:56.344265 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:56.346421 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:56.350539 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.350544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.350547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.350922 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.351412 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.351416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.351418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.351571 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.352380 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:56.352384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:56.352500 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 02:38:56.352660 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 02:38:56.352882 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260620 02:38:56.353540 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.337858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.0.97.61:33345)
0620 02:38:56.338655 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.338662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.338690 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:38:56.338746 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.338752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.338989 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.339249 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.340139 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.341330 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.342613 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.343170 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.343311 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.346682 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.346697 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.346726 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.346757 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.350383 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.351106 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.351113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.351116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.351311 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.351719 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.351723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.351725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.352258 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.352427 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.352827 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.353064 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":611,"thread_start_us":436,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 02:38:56.355767 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34223
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36073,127.0.97.61:33345,127.0.97.60:34223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:38:56.356101 14034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:38:56.357429 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.356885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44590)
0620 02:38:56.357098 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.357273 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 02:38:56.357738 13983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
I20260620 02:38:56.359385 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.359034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44602)
0620 02:38:56.359185 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.359277 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 02:38:56.359606 14034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34223: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
I20260620 02:38:56.408946 14034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } attempt: 1
I20260620 02:38:56.410248 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.409772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44606)
0620 02:38:56.409967 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.410089 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 02:38:56.411365 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.410973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44620)
0620 02:38:56.411170 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.411245 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 02:38:56.411653 14034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34223: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
I20260620 02:38:56.461954 13960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } attempt: 5
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:38:56.478477 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.462773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54726)
0620 02:38:56.462974 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.462996 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:56.463116 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:56.463522 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:56.463526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:56.463982 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:56.464167 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.464177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.465666 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.465671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.466806 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.466830 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.467954 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.467960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.468151 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.468837 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:56.468857 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:56.470935 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:56.475459 (+ 4524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.475464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.475467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.476051 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.476570 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.476574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.476577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.476768 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.477486 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:56.477492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:56.477678 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 02:38:56.477860 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 02:38:56.478153 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 02:38:56.478582 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.462866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54726 (local address 127.0.97.61:33345)
0620 02:38:56.463058 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.463065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.463090 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:38:56.463271 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.463277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.463358 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.463482 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.464341 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.465449 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.466999 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.467698 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.467868 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.471181 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.471188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.471193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.471245 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.475222 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.476225 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.476234 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.476238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.476397 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.476898 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.476903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.476906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.477300 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.477461 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.478020 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.478271 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:38:56.479916 14034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } attempt: 2
I20260620 02:38:56.479935 13960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:38:56.481015 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.480473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44626)
0620 02:38:56.480730 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.480836 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:38:56.481238 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.480760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44634)
0620 02:38:56.480953 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.481048 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 02:38:56.482350 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.481984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44650)
0620 02:38:56.482138 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.482229 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 02:38:56.482481 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.482022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44666)
0620 02:38:56.482228 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.482316 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260620 02:38:56.482684 14034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34223: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
W20260620 02:38:56.482697 13960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34223: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
I20260620 02:38:56.533964 13960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } attempt: 1
I20260620 02:38:56.535367 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.534865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44670)
0620 02:38:56.535080 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.535174 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 02:38:56.536595 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.536242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44676)
0620 02:38:56.536409 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.536481 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:38:56.536836 13960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34223: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
I20260620 02:38:56.573968 14034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } attempt: 3
I20260620 02:38:56.575307 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.574699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44688)
0620 02:38:56.574961 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.575079 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 02:38:56.576574 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.576068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44698)
0620 02:38:56.576311 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.576414 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260620 02:38:56.576820 14034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34223: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
I20260620 02:38:56.605085 13960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } attempt: 2
I20260620 02:38:56.607156 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.606390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44706)
0620 02:38:56.606763 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.606925 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 02:38:56.608943 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.608429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44718)
0620 02:38:56.608661 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.608764 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 02:38:56.609378 13960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34223: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
I20260620 02:38:56.715794 13960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } attempt: 3
I20260620 02:38:56.717227 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.716737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44720)
0620 02:38:56.716944 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.717049 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 02:38:56.718564 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.718099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44724)
0620 02:38:56.718351 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.718433 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260620 02:38:56.718443 14042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:38:56.718809 14042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:38:56.718866 13960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34223: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
W20260620 02:38:56.718990 14042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 02:38:56.720041 14034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } attempt: 4
I20260620 02:38:56.721223 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.720680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44734)
0620 02:38:56.720925 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.721028 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 02:38:56.722352 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.721877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44750)
0620 02:38:56.722058 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.722189 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 02:38:56.722633 14034 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34223: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34223: connect: Connection refused (error 111)
W20260620 02:38:56.727546 14042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:38:56.727680 14042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:38:56.727726 14042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:38:56.727777 14042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:38:56.727891 14042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:38:56.738617 14042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36073,127.0.97.61:33345,127.0.97.60:34223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:38:56.740285 14042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:38:56.741886 14042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:38:56.755152 14047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:56.755208 14048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:38:56.755282 14050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:38:56.756048 14042 server_base.cc:1034] running on GCE node
I20260620 02:38:56.756855 14042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:38:56.758469 14042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:38:56.759749 14042 hybrid_clock.cc:648] HybridClock initialized: now 1781923136759700 us; error 50 us; skew 500 ppm
I20260620 02:38:56.760270 14042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:38:56.763916 14042 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 02:38:56.765739 14042 webserver.cc:460] Webserver started at http://127.0.97.60:37183/ using document root <none> and password file <none>
I20260620 02:38:56.766523 14042 fs_manager.cc:359] Metadata directory not provided
I20260620 02:38:56.766690 14042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:38:56.767038 14042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:38:56.770316 14042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/data/instance:
uuid: "4d2ff211e87c4d76b4f1db8292510cb1"
format_stamp: "Formatted at 2026-06-20 02:38:56 on dist-test-slave-rpg6"
server_key: "e8408b2eb8b57b48cc2da56c5c9ec24d"
server_key_iv: "82d8412f2d02188de357a48d7101e728"
server_key_version: "encryptionkey@0"
I20260620 02:38:56.771148 14042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/wal/instance:
uuid: "4d2ff211e87c4d76b4f1db8292510cb1"
format_stamp: "Formatted at 2026-06-20 02:38:56 on dist-test-slave-rpg6"
server_key: "e8408b2eb8b57b48cc2da56c5c9ec24d"
server_key_iv: "82d8412f2d02188de357a48d7101e728"
server_key_version: "encryptionkey@0"
I20260620 02:38:56.776875 14042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:38:56.780578 14057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:38:56.782038 14042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:38:56.782258 14042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/wal
uuid: "4d2ff211e87c4d76b4f1db8292510cb1"
format_stamp: "Formatted at 2026-06-20 02:38:56 on dist-test-slave-rpg6"
server_key: "e8408b2eb8b57b48cc2da56c5c9ec24d"
server_key_iv: "82d8412f2d02188de357a48d7101e728"
server_key_version: "encryptionkey@0"
I20260620 02:38:56.782486 14042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:38:56.816402 14042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:38:56.821658 14042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:38:56.822047 14042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:38:56.844508 14042 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34223
I20260620 02:38:56.844521 14108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34223 every 8 connection(s)
I20260620 02:38:56.846124 14042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:38:56.850625 14109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:38:56.851923 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 14042
I20260620 02:38:56.852247 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781922915737669-388-0/minicluster-data/master-2/wal/instance
I20260620 02:38:56.852737 388 external_mini_cluster.cc:1428] Setting key c26aa104929f5162e6078f4676b4e867
I20260620 02:38:56.861246 14109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:38:56.880630 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.854950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.0.97.60:34223)
0620 02:38:56.856059 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.856072 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.856125 (+ 53us) server_negotiation.cc:407] Connection header received
0620 02:38:56.857852 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.857878 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.858379 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.858791 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.860864 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.862412 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.864323 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.865607 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.866495 (+ 888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.870993 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.871006 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.871025 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.871078 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.875205 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.876521 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.876534 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.876546 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.876755 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.877296 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.877302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.877306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.878087 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.878334 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.878930 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.879353 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":479,"threads_started":1}
I20260620 02:38:56.882440 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.864482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36073 (local address 127.0.0.1:44830)
0620 02:38:56.865363 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.865401 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:56.865618 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:56.866040 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:56.866047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:56.866909 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:56.867288 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.867301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.868413 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.868419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.869769 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.869782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.870993 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.871001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.871251 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.872119 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:56.872145 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:56.875235 (+ 3090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:56.879450 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.879463 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.879471 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.880060 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.880498 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.880503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.880507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.880668 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.881354 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:56.881362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:56.881566 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 02:38:56.881768 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 02:38:56.882129 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":495,"threads_started":1}
I20260620 02:38:56.882596 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.864406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.0.97.62:36073)
0620 02:38:56.864959 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.864968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.865512 (+ 544us) server_negotiation.cc:407] Connection header received
0620 02:38:56.865759 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.865766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.865847 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.865967 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.867462 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.868255 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.869924 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.870641 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.870798 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.875529 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.875540 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.875547 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.875603 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.879296 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.880218 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.880225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.880228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.880373 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.880828 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.880834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.880837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.881208 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.881337 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.881874 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.882099 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":266,"threads_started":1}
I20260620 02:38:56.884140 14109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:38:56.896126 13960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } attempt: 4
I20260620 02:38:56.899817 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.884948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33345 (local address 127.0.0.1:54740)
0620 02:38:56.885117 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.885137 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:56.885225 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:56.885563 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:56.885567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:56.886134 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:56.886357 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.886371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.887640 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.887646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.888609 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.888622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.889503 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.889509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.889778 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.890478 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:56.890504 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:56.892890 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:56.896564 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.896570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.896574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.897342 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.897811 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.897816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.897819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.898112 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.898761 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:56.898766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:56.898935 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 02:38:56.899088 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 02:38:56.899405 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 02:38:56.899958 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.885086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.0.97.61:33345)
0620 02:38:56.885290 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.885294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.885310 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:38:56.885358 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.885361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.885416 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.885495 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.886559 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.887474 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.888762 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.889356 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.889453 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.893131 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.893137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.893141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.893179 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.896426 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.897472 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.897480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.897484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.897686 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.898237 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.898243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.898246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.898651 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.898813 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.899207 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.899453 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 02:38:56.900945 14109 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:38:56.901387 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.884402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.0.97.62:36073)
0620 02:38:56.884598 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.884605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.884720 (+ 115us) server_negotiation.cc:407] Connection header received
0620 02:38:56.884938 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.884943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.885033 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.886857 (+ 1824us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.886967 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.887701 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.889181 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.889698 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.889824 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.894474 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.894479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.894483 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.894518 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.897793 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.898695 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.898703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.898708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.898861 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.899426 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.899431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.899433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.899793 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.900138 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.900773 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.901032 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:38:56.914094 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.896862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.0.97.60:34223)
0620 02:38:56.897022 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.897028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.897172 (+ 144us) server_negotiation.cc:407] Connection header received
0620 02:38:56.897349 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.897357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.897444 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.897560 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.898511 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.899620 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.901729 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.902426 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.902588 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.906633 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.906639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.906644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.906692 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.910688 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.911642 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.911648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.911651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.911802 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.912325 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.912333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.912337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.912745 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.912907 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.913395 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.913675 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 02:38:56.914105 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.896803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44756)
0620 02:38:56.897060 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.897079 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:56.897219 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:56.897613 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:56.897618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:56.898115 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:56.898355 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.898368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.899778 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.899783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.901170 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.901186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.902621 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.902629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.902831 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.903682 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:56.903707 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:56.906408 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:56.910868 (+ 4460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.910878 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.910884 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.911488 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.911966 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.911972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.911977 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.912172 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.912909 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:56.912915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:56.913085 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 02:38:56.913279 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 02:38:56.913657 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":1}
Jun 20 02:38:56 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:38:56.920814 13960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c: Bootstrap starting.
I20260620 02:38:56.921520 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.901970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44772)
0620 02:38:56.902166 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:56.902190 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:56.902294 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:56.903945 (+ 1651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:56.903950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:56.904452 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:56.904646 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.904664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.906421 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.906425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.907940 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:56.907958 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.909231 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.909239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:56.909475 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.910543 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:56.910575 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:56.912997 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:56.917141 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.917146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.917150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.917665 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.919457 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:56.919462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:56.919466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:56.919643 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:56.920376 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:56.920382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:56.920547 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 02:38:56.920693 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 02:38:56.921059 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 02:38:56.921991 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.902250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.0.97.60:34223)
0620 02:38:56.902996 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.903003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.903027 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:38:56.903078 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.903083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.903159 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.903269 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.904843 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.906275 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.908111 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.908849 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.909001 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.913212 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.913216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.913220 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.913276 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.916984 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.919103 (+ 2119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.919121 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.919124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.919293 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.919796 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.919803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.919810 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.920232 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.920396 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.920851 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.921264 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":620,"thread_start_us":957,"threads_started":1}
I20260620 02:38:56.922907 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.903995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.0.97.61:33345)
0620 02:38:56.904174 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.904180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.904268 (+ 88us) server_negotiation.cc:407] Connection header received
0620 02:38:56.904445 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.904451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.904528 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.904632 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.906416 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.907302 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.909220 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.909678 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.909858 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.916031 (+ 6173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.916036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.916040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.916080 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.918892 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.919905 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.919911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.919914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.920080 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.920584 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.920588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.920590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.921126 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.921428 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.922261 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.922538 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:38:56.925290 13960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c: Neither blocks nor log segments found. Creating new log.
I20260620 02:38:56.926702 13960 log.cc:824] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:38:56.928010 14109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1: Bootstrap starting.
I20260620 02:38:56.929471 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:38:56.930967 13960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c: No bootstrap required, opened a new log
I20260620 02:38:56.932309 14109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1: Neither blocks nor log segments found. Creating new log.
I20260620 02:38:56.933727 14109 log.cc:824] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:38:56.937726 14109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1: No bootstrap required, opened a new log
I20260620 02:38:56.943044 13960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } }
I20260620 02:38:56.943534 13960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:38:56.943660 13960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e41de5bb3b64703ad470af1d519368c, State: Initialized, Role: FOLLOWER
I20260620 02:38:56.944612 13960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } }
I20260620 02:38:56.951753 14124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } } }
I20260620 02:38:56.952243 14124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:38:56.953728 14109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } }
I20260620 02:38:56.954257 14109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:38:56.954347 13960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:38:56.954412 14109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2ff211e87c4d76b4f1db8292510cb1, State: Initialized, Role: FOLLOWER
I20260620 02:38:56.965487 14109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } }
W20260620 02:38:56.973179 13914 tablet.cc:2369] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:38:56.975049 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.938729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.0.97.60:34223)
0620 02:38:56.938967 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.938979 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.949766 (+ 10787us) server_negotiation.cc:407] Connection header received
0620 02:38:56.950056 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.950066 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.950209 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.950367 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.953695 (+ 3328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.955168 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.958009 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.959000 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.959160 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.964856 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.964868 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.964876 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.964948 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.970128 (+ 5180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.971538 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.971548 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.971555 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.971779 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.972447 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.972456 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.972461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.972980 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.973336 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.974080 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.974478 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 02:38:56.976671 14127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } } }
I20260620 02:38:56.977347 14127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:38:56.980979 14109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:38:56.993587 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.938269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.0.97.62:36073)
0620 02:38:56.938486 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.938496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.955548 (+ 17052us) server_negotiation.cc:407] Connection header received
0620 02:38:56.955945 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.955954 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.956070 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.956166 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.959670 (+ 3504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.961316 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.966564 (+ 5248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.967734 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.967923 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.979087 (+ 11164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.979098 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.979106 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.979157 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.987485 (+ 8328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.989042 (+ 1557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.989051 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.989058 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.989291 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.989954 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.989962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.989968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.990480 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:56.990944 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:56.992608 (+ 1664us) server_negotiation.cc:299] Negotiation successful
0620 02:38:56.992987 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:38:57.001477 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:56.955698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.0.97.61:33345)
0620 02:38:56.955934 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:56.955941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:56.978879 (+ 22938us) server_negotiation.cc:407] Connection header received
0620 02:38:56.979166 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:56.979171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:56.979266 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:56.979348 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:56.981444 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.982978 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.985663 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:56.986692 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:56.986850 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:56.992073 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:56.992085 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:56.992092 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:56.992170 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:56.996912 (+ 4742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.998281 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.998291 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.998298 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.998544 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:56.999168 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:56.999176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:56.999181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:56.999672 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:57.000046 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:57.000584 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 02:38:57.000942 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 02:38:57.018918 14145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:38:57.019115 14145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:38:57.023658 14034 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } attempt: 5
W20260620 02:38:57.023769 14148 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:38:57.023940 14148 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:38:57.019886 13893 master_runner.cc:186] Error getting master registration for 127.0.97.61:33345: OK, instance_id { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" instance_seqno: 1781923136286962 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:38:57.031648 13893 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 02:38:57 dist-test-slave-rpg6 krb5kdc[13880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:38:57.026219 14042 master_runner.cc:186] Error getting master registration for 127.0.97.61:33345: OK, instance_id { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" instance_seqno: 1781923136286962 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:38:57.038956 14042 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:38:57.046787 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:57.024961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34223 (local address 127.0.0.1:44802)
0620 02:38:57.025428 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 02:38:57.025449 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:38:57.025575 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:38:57.026595 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:38:57.026601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:38:57.027204 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:38:57.027428 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:57.027440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:57.031932 (+ 4492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:57.031940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:57.033468 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:38:57.033488 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:38:57.034762 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:38:57.034769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:38:57.034952 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:57.035671 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:38:57.035698 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:38:57.038685 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:38:57.043504 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:57.043509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:57.043514 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:57.044117 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:57.044692 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:38:57.044698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:38:57.044701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:38:57.044875 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:38:57.045656 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:38:57.045661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:38:57.045843 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 02:38:57.046015 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 02:38:57.046359 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335}
I20260620 02:38:57.046952 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:38:57.025926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.0.97.60:34223)
0620 02:38:57.026115 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:38:57.026122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:38:57.026141 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:38:57.026201 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:38:57.026207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:38:57.026291 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:38:57.026393 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:38:57.027612 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:57.028864 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:57.033674 (+ 4810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:38:57.034457 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:38:57.034587 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:38:57.039008 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:38:57.039019 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:38:57.039025 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:38:57.039071 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:38:57.043282 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:57.044292 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:57.044299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:57.044303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:57.044507 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:38:57.045030 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:38:57.045037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:38:57.045041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:38:57.045504 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:38:57.045713 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 02:38:57.046197 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 02:38:57.046479 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:38:57.054133 14034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010: Bootstrap starting.
I20260620 02:38:57.059037 14034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010: Neither blocks nor log segments found. Creating new log.
I20260620 02:38:57.060873 14034 log.cc:824] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010: Log is configured to *not* fsync() on all Append() calls
I20260620 02:38:57.065171 14034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010: No bootstrap required, opened a new log
W20260620 02:38:57.074946 14063 tablet.cc:2369] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:38:57.078765 14034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } }
I20260620 02:38:57.079305 14034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:38:57.079458 14034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7047a6d4fc3648a093fdc7b006a34010, State: Initialized, Role: FOLLOWER
I20260620 02:38:57.080330 14034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } }
I20260620 02:38:57.082763 14152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } } }
I20260620 02:38:57.083261 14152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:38:57.084434 14034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:38:57.106904 13967 master_runner.cc:263] No leader master found from master 7047a6d4fc3648a093fdc7b006a34010
I20260620 02:38:57.107075 13967 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 02:38:57.118916 14163 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:38:57.119118 14163 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:38:57.174245 14152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:38:57.174660 14152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } }
I20260620 02:38:57.176741 14152 leader_election.cc:288] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e41de5bb3b64703ad470af1d519368c (127.0.97.62:36073), 4d2ff211e87c4d76b4f1db8292510cb1 (127.0.97.60:34223)
I20260620 02:38:57.178085 13935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7047a6d4fc3648a093fdc7b006a34010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e41de5bb3b64703ad470af1d519368c" is_pre_election: true
I20260620 02:38:57.178867 13935 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7047a6d4fc3648a093fdc7b006a34010 in term 0.
I20260620 02:38:57.179739 13986 leader_election.cc:302] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e41de5bb3b64703ad470af1d519368c, 7047a6d4fc3648a093fdc7b006a34010; no voters:
I20260620 02:38:57.181152 14084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7047a6d4fc3648a093fdc7b006a34010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" is_pre_election: true
I20260620 02:38:57.181915 14084 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7047a6d4fc3648a093fdc7b006a34010 in term 0.
I20260620 02:38:57.183051 14152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:38:57.183252 14152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:38:57.183372 14152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:38:57.192461 14152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } }
I20260620 02:38:57.196143 13935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7047a6d4fc3648a093fdc7b006a34010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e41de5bb3b64703ad470af1d519368c"
I20260620 02:38:57.196475 13935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:38:57.200817 13935 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7047a6d4fc3648a093fdc7b006a34010 in term 1.
I20260620 02:38:57.201529 13986 leader_election.cc:302] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e41de5bb3b64703ad470af1d519368c, 7047a6d4fc3648a093fdc7b006a34010; no voters:
I20260620 02:38:57.206594 14084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7047a6d4fc3648a093fdc7b006a34010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d2ff211e87c4d76b4f1db8292510cb1"
I20260620 02:38:57.206943 14084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:38:57.211014 14084 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7047a6d4fc3648a093fdc7b006a34010 in term 1.
I20260620 02:38:57.212152 14152 leader_election.cc:288] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [CANDIDATE]: Term 1 election: Requested vote from peers 5e41de5bb3b64703ad470af1d519368c (127.0.97.62:36073), 4d2ff211e87c4d76b4f1db8292510cb1 (127.0.97.60:34223)
I20260620 02:38:57.212446 14152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:38:57.212666 14152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [term 1 LEADER]: Becoming Leader. State: Replica: 7047a6d4fc3648a093fdc7b006a34010, State: Running, Role: LEADER
I20260620 02:38:57.219275 14152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } }
I20260620 02:38:57.232136 14165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7047a6d4fc3648a093fdc7b006a34010. Latest consensus state: current_term: 1 leader_uuid: "7047a6d4fc3648a093fdc7b006a34010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } } }
I20260620 02:38:57.232594 14165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [sys.catalog]: This master's current role is: LEADER
I20260620 02:38:57.239667 14167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:38:57.242522 14167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:38:57.264119 14084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [term 1 FOLLOWER]: Refusing update from remote peer 7047a6d4fc3648a093fdc7b006a34010: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:38:57.265259 14165 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:38:57.272917 13935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [term 1 FOLLOWER]: Refusing update from remote peer 7047a6d4fc3648a093fdc7b006a34010: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 02:38:57.275967 13988 tablet.cc:2369] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:38:57.279055 14165 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:38:57.313905 14124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7047a6d4fc3648a093fdc7b006a34010. Latest consensus state: current_term: 1 leader_uuid: "7047a6d4fc3648a093fdc7b006a34010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } } }
I20260620 02:38:57.314296 14124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:38:57.317755 14127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7047a6d4fc3648a093fdc7b006a34010. Latest consensus state: current_term: 1 leader_uuid: "7047a6d4fc3648a093fdc7b006a34010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } } }
I20260620 02:38:57.318068 14127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:38:57.321735 14168 mvcc.cc:205] Tried to move back new op lower bound from 7298757170219425792 to 7298757170066325504. Current Snapshot: MvccSnapshot[applied={T|T < 7298757170219425792}]
I20260620 02:38:57.329838 14169 mvcc.cc:205] Tried to move back new op lower bound from 7298757170219425792 to 7298757170066325504. Current Snapshot: MvccSnapshot[applied={T|T < 7298757170219425792}]
I20260620 02:38:57.332077 14152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7047a6d4fc3648a093fdc7b006a34010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } } }
I20260620 02:38:57.332448 14152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [sys.catalog]: This master's current role is: LEADER
I20260620 02:38:57.332762 14152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7047a6d4fc3648a093fdc7b006a34010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } } }
I20260620 02:38:57.333007 14152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010 [sys.catalog]: This master's current role is: LEADER
I20260620 02:38:57.333194 14127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7047a6d4fc3648a093fdc7b006a34010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } } }
I20260620 02:38:57.333477 14127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:38:57.339872 14167 catalog_manager.cc:1347] Generated new cluster ID: adb1a00d23c14967b1da85f8cc090e48
I20260620 02:38:57.340014 14167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:38:57.349432 14124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7047a6d4fc3648a093fdc7b006a34010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e41de5bb3b64703ad470af1d519368c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36073 } } peers { permanent_uuid: "7047a6d4fc3648a093fdc7b006a34010" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33345 } } peers { permanent_uuid: "4d2ff211e87c4d76b4f1db8292510cb1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34223 } } }
I20260620 02:38:57.349721 14124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:38:57.386423 14167 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:38:57.388664 14167 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:38:57.417119 14167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7047a6d4fc3648a093fdc7b006a34010: Generated new TSK 0
I20260620 02:38:57.418442 14167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:38:58.021246 14145 catalog_manager.cc:1259] Loaded cluster ID: adb1a00d23c14967b1da85f8cc090e48
I20260620 02:38:58.021431 14145 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c: loading cluster ID for follower catalog manager: success
I20260620 02:38:58.025501 14145 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c: acquiring CA information for follower catalog manager: success
I20260620 02:38:58.026468 14148 catalog_manager.cc:1259] Loaded cluster ID: adb1a00d23c14967b1da85f8cc090e48
I20260620 02:38:58.026572 14148 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1: loading cluster ID for follower catalog manager: success
I20260620 02:38:58.028240 14145 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e41de5bb3b64703ad470af1d519368c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:38:58.029362 14148 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1: acquiring CA information for follower catalog manager: success
I20260620 02:38:58.031296 14148 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d2ff211e87c4d76b4f1db8292510cb1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:38:58.543112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 13893
I20260620 02:38:58.545186 13955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:38:58.658589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 13893
I20260620 02:38:58.681738 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 13967
I20260620 02:38:58.683159 14029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:38:58.797706 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 13967
I20260620 02:38:58.820122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 14042
I20260620 02:38:58.821502 14104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:38:58.932643 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 14042
2026-06-20T02:38:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5073 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:38:59 dist-test-slave-rpg6 krb5kdc[14184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:38:59 dist-test-slave-rpg6 krb5kdc[14184](info): set up 2 sockets
Jun 20 02:38:59 dist-test-slave-rpg6 krb5kdc[14184](info): commencing operation
krb5kdc: starting...
W20260620 02:39:00.072667 388 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:39:00 dist-test-slave-rpg6 krb5kdc[14184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:39:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:39:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:39:00.328640 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37629
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:00.694110 14197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:00.694456 14197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:00.694651 14197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:00.702905 14197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:39:00.703063 14197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:00.703120 14197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:00.703178 14197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:39:00.703250 14197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:39:00.714570 14197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:00.716039 14197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:00.717435 14197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:00.730547 14202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:00.730619 14203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:00.730850 14205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:00.731645 14197 server_base.cc:1034] running on GCE node
I20260620 02:39:00.732602 14197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:00.734266 14197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:00.735574 14197 hybrid_clock.cc:648] HybridClock initialized: now 1781923140735513 us; error 46 us; skew 500 ppm
I20260620 02:39:00.736089 14197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:00 dist-test-slave-rpg6 krb5kdc[14184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:00.745002 14197 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:39:00.746938 14197 webserver.cc:460] Webserver started at http://127.0.97.62:44281/ using document root <none> and password file <none>
I20260620 02:39:00.747758 14197 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:00.747974 14197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:00.748354 14197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:39:00.751622 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "52ffd4a6406e47e5aef799667df063f7"
format_stamp: "Formatted at 2026-06-20 02:39:00 on dist-test-slave-rpg6"
server_key: "10a540e3c7a0e7f6c662e54e9f3cdef2"
server_key_iv: "95ab72cc486637908eacd5af06bafac6"
server_key_version: "encryptionkey@0"
I20260620 02:39:00.752622 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "52ffd4a6406e47e5aef799667df063f7"
format_stamp: "Formatted at 2026-06-20 02:39:00 on dist-test-slave-rpg6"
server_key: "10a540e3c7a0e7f6c662e54e9f3cdef2"
server_key_iv: "95ab72cc486637908eacd5af06bafac6"
server_key_version: "encryptionkey@0"
I20260620 02:39:00.758498 14197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:39:00.762593 14212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:00.764374 14197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:39:00.764611 14197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "52ffd4a6406e47e5aef799667df063f7"
format_stamp: "Formatted at 2026-06-20 02:39:00 on dist-test-slave-rpg6"
server_key: "10a540e3c7a0e7f6c662e54e9f3cdef2"
server_key_iv: "95ab72cc486637908eacd5af06bafac6"
server_key_version: "encryptionkey@0"
I20260620 02:39:00.764860 14197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:00.791502 14197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:00.796343 14197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:00.796705 14197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:00.817478 14197 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37629
I20260620 02:39:00.817507 14263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37629 every 8 connection(s)
I20260620 02:39:00.819105 14197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:39:00.823352 14264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:00.826062 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 14197
I20260620 02:39:00.826311 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:39:00.827175 388 external_mini_cluster.cc:1428] Setting key 3a8f6ac9ed8acddcec48cf64b516f4d8
I20260620 02:39:00.838176 14264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7: Bootstrap starting.
I20260620 02:39:00.842859 14264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7: Neither blocks nor log segments found. Creating new log.
Jun 20 02:39:00 dist-test-slave-rpg6 krb5kdc[14184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:00.844364 14264 log.cc:824] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:00.847668 14264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7: No bootstrap required, opened a new log
I20260620 02:39:00.853894 14267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:00.829375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58426 (local address 127.0.97.62:37629)
0620 02:39:00.830401 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:00.830417 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:00.830469 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:39:00.832421 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:00.832448 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:00.832892 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:00.833253 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:00.835492 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:00.837082 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:00.839063 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:00.839947 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:00.840185 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:00.845596 (+ 5411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:00.845618 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:00.845641 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:00.845683 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:00.849108 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:00.849996 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:00.850003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:00.850010 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:00.850210 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:00.850671 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:00.850677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:00.850686 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:00.851308 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:00.851624 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:00.852212 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 02:39:00.852686 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":498,"threads_started":1}
I20260620 02:39:00.856235 14264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ffd4a6406e47e5aef799667df063f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37629 } }
I20260620 02:39:00.856745 14264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:00.856865 14264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ffd4a6406e47e5aef799667df063f7, State: Initialized, Role: FOLLOWER
I20260620 02:39:00.857514 14264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ffd4a6406e47e5aef799667df063f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37629 } }
I20260620 02:39:00.857803 14264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:39:00.857973 14264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:39:00.858174 14264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:00.858559 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:39:00.861840 14264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ffd4a6406e47e5aef799667df063f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37629 } }
I20260620 02:39:00.862591 14264 leader_election.cc:302] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52ffd4a6406e47e5aef799667df063f7; no voters:
I20260620 02:39:00.863358 14264 leader_election.cc:288] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:39:00.863535 14269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:39:00.864591 14269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [term 1 LEADER]: Becoming Leader. State: Replica: 52ffd4a6406e47e5aef799667df063f7, State: Running, Role: LEADER
I20260620 02:39:00.865343 14269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ffd4a6406e47e5aef799667df063f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37629 } }
I20260620 02:39:00.865859 14264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:39:00.868757 14271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ffd4a6406e47e5aef799667df063f7. Latest consensus state: current_term: 1 leader_uuid: "52ffd4a6406e47e5aef799667df063f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ffd4a6406e47e5aef799667df063f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37629 } } }
I20260620 02:39:00.868717 14270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52ffd4a6406e47e5aef799667df063f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ffd4a6406e47e5aef799667df063f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37629 } } }
I20260620 02:39:00.869321 14271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:00.869360 14270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:00.871451 14277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:39:00.877331 14277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:39:00.887923 14277 catalog_manager.cc:1347] Generated new cluster ID: c2944383924b41de8206b69d5f8c0cf1
I20260620 02:39:00.888104 14277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:39:00.901094 14277 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:39:00.903055 14277 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:39:00.914080 14277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52ffd4a6406e47e5aef799667df063f7: Generated new TSK 0
I20260620 02:39:00.915258 14277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:39:00.976099 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 14197
I20260620 02:39:00.977562 14259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:01.085562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 14197
2026-06-20T02:39:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2151 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:39:01 dist-test-slave-rpg6 krb5kdc[14292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:39:01 dist-test-slave-rpg6 krb5kdc[14292](info): set up 2 sockets
Jun 20 02:39:01 dist-test-slave-rpg6 krb5kdc[14292](info): commencing operation
krb5kdc: starting...
W20260620 02:39:02.216823 388 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:39:02 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:39:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:39:02.494303 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37319
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:02.858948 14305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:02.859624 14305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:02.859920 14305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:02.867812 14305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:39:02.867996 14305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:02.868046 14305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:02.868136 14305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:39:02.868206 14305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:39:02.879024 14305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:02.880535 14305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:02.881954 14305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:02.896512 14310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:02.896461 14313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:02.896483 14311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:02.897591 14305 server_base.cc:1034] running on GCE node
I20260620 02:39:02.898613 14305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:02.900764 14305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:02.902056 14305 hybrid_clock.cc:648] HybridClock initialized: now 1781923142901989 us; error 62 us; skew 500 ppm
I20260620 02:39:02.902501 14305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:02 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:02.906212 14305 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:39:02.908118 14305 webserver.cc:460] Webserver started at http://127.0.97.62:45153/ using document root <none> and password file <none>
I20260620 02:39:02.908931 14305 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:02.909103 14305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:02.909451 14305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:39:02.912915 14305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3"
format_stamp: "Formatted at 2026-06-20 02:39:02 on dist-test-slave-rpg6"
server_key: "3bf9fce644a60c2a9b094ee90c375fd3"
server_key_iv: "fdaba8ca6946450cc5e27e42abfeddbd"
server_key_version: "encryptionkey@0"
I20260620 02:39:02.913830 14305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3"
format_stamp: "Formatted at 2026-06-20 02:39:02 on dist-test-slave-rpg6"
server_key: "3bf9fce644a60c2a9b094ee90c375fd3"
server_key_iv: "fdaba8ca6946450cc5e27e42abfeddbd"
server_key_version: "encryptionkey@0"
I20260620 02:39:02.919804 14305 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 02:39:02.923885 14320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:02.925619 14305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:39:02.925945 14305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3"
format_stamp: "Formatted at 2026-06-20 02:39:02 on dist-test-slave-rpg6"
server_key: "3bf9fce644a60c2a9b094ee90c375fd3"
server_key_iv: "fdaba8ca6946450cc5e27e42abfeddbd"
server_key_version: "encryptionkey@0"
I20260620 02:39:02.926191 14305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:02.939687 14305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:02.944710 14305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:02.945047 14305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:02.966224 14305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37319
I20260620 02:39:02.966254 14371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37319 every 8 connection(s)
I20260620 02:39:02.967813 14305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:39:02.971048 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 14305
I20260620 02:39:02.971324 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:39:02.972087 388 external_mini_cluster.cc:1428] Setting key 11d3d6cc6e8c2600b12364c3261d75f9
I20260620 02:39:02.972153 14372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:02.988744 14372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Bootstrap starting.
Jun 20 02:39:02 dist-test-slave-rpg6 krb5kdc[14292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:02.993539 14372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:02.995151 14372 log.cc:824] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:02.999413 14372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: No bootstrap required, opened a new log
I20260620 02:39:03.000442 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:02.974241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54172 (local address 127.0.97.62:37319)
0620 02:39:02.975102 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:02.975117 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:02.975158 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:39:02.977330 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:02.977356 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:02.977806 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:02.978184 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:02.980305 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:02.981758 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:02.983727 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:02.984760 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:02.984925 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:02.990242 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:02.990264 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:02.990292 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:02.990349 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:02.995108 (+ 4759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:02.996245 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:02.996254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:02.996261 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:02.996453 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:02.996954 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:02.996958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:02.996960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:02.997713 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:02.998007 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:02.998434 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 02:39:02.998940 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":419,"threads_started":1}
I20260620 02:39:03.009944 14372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:03.010396 14372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:03.010483 14372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ebf31f3bb34d23a0afec81d5eb1fc3, State: Initialized, Role: FOLLOWER
I20260620 02:39:03.011286 14372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:03.011727 14372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:39:03.011905 14372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:39:03.012120 14372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:03.014529 14372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:03.015297 14372 leader_election.cc:302] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52ebf31f3bb34d23a0afec81d5eb1fc3; no voters:
I20260620 02:39:03.015365 14377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:39:03.015726 14377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:03.016306 14372 leader_election.cc:288] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:39:03.016675 14378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:39:03.017521 14377 leader_election.cc:302] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52ebf31f3bb34d23a0afec81d5eb1fc3; no voters:
I20260620 02:39:03.017871 14378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 1 LEADER]: Becoming Leader. State: Replica: 52ebf31f3bb34d23a0afec81d5eb1fc3, State: Running, Role: LEADER
I20260620 02:39:03.018299 14377 leader_election.cc:288] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20260620 02:39:03.018577 14378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:03.019150 14372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:39:03.021173 14379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } } }
I20260620 02:39:03.022027 14379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:03.021903 14380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ebf31f3bb34d23a0afec81d5eb1fc3. Latest consensus state: current_term: 1 leader_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } } }
I20260620 02:39:03.022259 14380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:03.023020 14386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:39:03.029202 14386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:39:03.039533 14386 catalog_manager.cc:1347] Generated new cluster ID: 8dd9974dad7b466989fa5033525b38ee
I20260620 02:39:03.039705 14386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:39:03.066876 14386 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:39:03.069116 14386 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:39:03.084482 14386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Generated new TSK 0
I20260620 02:39:03.085563 14386 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:39:03.157833 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37319
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:03.525738 14400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:03.526110 14400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:03.526278 14400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:03.534343 14400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:03.534482 14400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:03.534679 14400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:39:03.545192 14400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:03.546773 14400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:03.548260 14400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:03.562564 14405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:03.562511 14406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:03.562587 14408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:03.563699 14400 server_base.cc:1034] running on GCE node
I20260620 02:39:03.564456 14400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:03.565634 14400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:03.566893 14400 hybrid_clock.cc:648] HybridClock initialized: now 1781923143566846 us; error 40 us; skew 500 ppm
I20260620 02:39:03.567329 14400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:03 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:03.570981 14400 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:39:03.572685 14400 webserver.cc:460] Webserver started at http://127.0.97.1:35623/ using document root <none> and password file <none>
I20260620 02:39:03.573417 14400 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:03.573527 14400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:03.573788 14400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:39:03.577076 14400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "7175767fdecd49e3a35b46f248578b8f"
format_stamp: "Formatted at 2026-06-20 02:39:03 on dist-test-slave-rpg6"
server_key: "d6a9aa1b087ff9bff18bb84d91aebd45"
server_key_iv: "048e748175302dad63494cf3fb5bb21c"
server_key_version: "encryptionkey@0"
I20260620 02:39:03.577901 14400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7175767fdecd49e3a35b46f248578b8f"
format_stamp: "Formatted at 2026-06-20 02:39:03 on dist-test-slave-rpg6"
server_key: "d6a9aa1b087ff9bff18bb84d91aebd45"
server_key_iv: "048e748175302dad63494cf3fb5bb21c"
server_key_version: "encryptionkey@0"
I20260620 02:39:03.583779 14400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:39:03.588256 14415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:03.590023 14400 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:39:03.590271 14400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "7175767fdecd49e3a35b46f248578b8f"
format_stamp: "Formatted at 2026-06-20 02:39:03 on dist-test-slave-rpg6"
server_key: "d6a9aa1b087ff9bff18bb84d91aebd45"
server_key_iv: "048e748175302dad63494cf3fb5bb21c"
server_key_version: "encryptionkey@0"
I20260620 02:39:03.590582 14400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:03.609736 14400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:03.614615 14400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:03.614961 14400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:03.616125 14400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:03.617784 14400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:03.617862 14400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:03.618003 14400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:03.618067 14400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:03.653930 14400 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36939
I20260620 02:39:03.654027 14527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36939 every 8 connection(s)
I20260620 02:39:03.655283 14400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:39:03.658293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 14400
I20260620 02:39:03.658501 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:39:03.659391 388 external_mini_cluster.cc:1428] Setting key fc8380312255d395dba19267bb84976f
Jun 20 02:39:03 dist-test-slave-rpg6 krb5kdc[14292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:03.676608 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:03.658054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33189 (local address 127.0.97.62:37319)
0620 02:39:03.658577 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:03.658585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:03.659719 (+ 1134us) server_negotiation.cc:407] Connection header received
0620 02:39:03.661940 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:03.661945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:03.662049 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:03.662170 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:03.663683 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:03.664497 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:03.666108 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:03.666520 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:03.666664 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:03.670508 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:03.670514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:03.670517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:03.670555 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:03.673300 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:03.674148 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:03.674159 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:03.674161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:03.674311 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:03.674774 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:03.674781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:03.674784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:03.675057 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:03.675225 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:03.675983 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 02:39:03.676237 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":259,"threads_started":1}
I20260620 02:39:03.677701 14531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:03.658692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:33189)
0620 02:39:03.659548 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:03.659600 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:03.661672 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:03.662418 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:03.662427 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:03.662939 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:03.663514 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:03.663530 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:03.664675 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:03.664691 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:03.665945 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:03.665957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:03.666773 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:03.666782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:03.667032 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:03.667746 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:03.667769 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:03.670243 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:03.673493 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:03.673505 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:03.673532 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:03.674005 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:03.674427 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:03.674432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:03.674436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:03.674649 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:03.675190 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:03.675199 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:03.675574 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 02:39:03.675891 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 02:39:03.676471 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":340,"threads_started":1}
I20260620 02:39:03.679404 14528 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37319
I20260620 02:39:03.679776 14528 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:03.680799 14528 heartbeater.cc:511] Master 127.0.97.62:37319 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:39:03.683249 14337 ts_manager.cc:194] Registered new tserver with Master: 7175767fdecd49e3a35b46f248578b8f (127.0.97.1:36939)
I20260620 02:39:03.684818 14337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:39:03.716960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37319
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:04.073788 14534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:04.074097 14534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:04.074265 14534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:04.081808 14534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:04.081943 14534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:04.082134 14534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:39:04.093079 14534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:04.094558 14534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:04.096189 14534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:04.111243 14539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:04.111169 14540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:04.111274 14542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:04.112124 14534 server_base.cc:1034] running on GCE node
I20260620 02:39:04.112890 14534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:04.113950 14534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:04.115231 14534 hybrid_clock.cc:648] HybridClock initialized: now 1781923144115156 us; error 65 us; skew 500 ppm
I20260620 02:39:04.115581 14534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:04 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:04.119109 14534 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:39:04.120951 14534 webserver.cc:460] Webserver started at http://127.0.97.2:39477/ using document root <none> and password file <none>
I20260620 02:39:04.121707 14534 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:04.121861 14534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:04.122195 14534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:39:04.125348 14534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "42db81b4b7aa474ea0042bd0f2d87fd4"
format_stamp: "Formatted at 2026-06-20 02:39:04 on dist-test-slave-rpg6"
server_key: "ecfffbb8962cdcc53e3b78c854ade989"
server_key_iv: "6d79f7d347c8cc6833d2cf169c7ca4fa"
server_key_version: "encryptionkey@0"
I20260620 02:39:04.126272 14534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "42db81b4b7aa474ea0042bd0f2d87fd4"
format_stamp: "Formatted at 2026-06-20 02:39:04 on dist-test-slave-rpg6"
server_key: "ecfffbb8962cdcc53e3b78c854ade989"
server_key_iv: "6d79f7d347c8cc6833d2cf169c7ca4fa"
server_key_version: "encryptionkey@0"
I20260620 02:39:04.131982 14534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 02:39:04.136025 14549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:04.137596 14534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:39:04.137898 14534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "42db81b4b7aa474ea0042bd0f2d87fd4"
format_stamp: "Formatted at 2026-06-20 02:39:04 on dist-test-slave-rpg6"
server_key: "ecfffbb8962cdcc53e3b78c854ade989"
server_key_iv: "6d79f7d347c8cc6833d2cf169c7ca4fa"
server_key_version: "encryptionkey@0"
I20260620 02:39:04.138128 14534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:04.162402 14534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:04.167100 14534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:04.167443 14534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:04.168519 14534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:04.170049 14534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:04.170194 14534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:04.170326 14534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:04.170387 14534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:04.207330 14534 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42081
I20260620 02:39:04.207432 14661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42081 every 8 connection(s)
I20260620 02:39:04.208770 14534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:39:04.213640 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 14534
I20260620 02:39:04.213856 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:39:04.214438 388 external_mini_cluster.cc:1428] Setting key c6d5d192bc06f6ef141152e27e87c3a3
Jun 20 02:39:04 dist-test-slave-rpg6 krb5kdc[14292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:04.234510 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:04.211744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57511 (local address 127.0.97.62:37319)
0620 02:39:04.212253 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:04.212258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:04.213477 (+ 1219us) server_negotiation.cc:407] Connection header received
0620 02:39:04.215922 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:04.215927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:04.216001 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:04.216107 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:04.218427 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.219367 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.222059 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.222503 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.222608 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:04.227362 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:04.227372 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:04.227380 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:04.227433 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:04.231114 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:04.232030 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:04.232034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:04.232036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:04.232179 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:04.232640 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:04.232651 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:04.232653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:04.232947 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:04.233109 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:04.233834 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 02:39:04.234103 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":266,"threads_started":1}
I20260620 02:39:04.235919 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:04.212365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:57511)
0620 02:39:04.213256 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:04.213313 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:04.215518 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:04.216437 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:04.216449 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:04.217282 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:04.218197 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:04.218227 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.219557 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.219567 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:04.221878 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:04.221900 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.222761 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.222771 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:04.223116 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:04.224013 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:04.224049 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:04.227072 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:04.231306 (+ 4234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:04.231315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:04.231331 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:04.231826 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:04.232297 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:04.232302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:04.232306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:04.232505 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:04.233095 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:04.233104 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:04.233461 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 02:39:04.233730 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 02:39:04.234315 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":337,"threads_started":1}
I20260620 02:39:04.237901 14662 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37319
I20260620 02:39:04.238267 14662 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:04.239086 14662 heartbeater.cc:511] Master 127.0.97.62:37319 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:39:04.240830 14337 ts_manager.cc:194] Registered new tserver with Master: 42db81b4b7aa474ea0042bd0f2d87fd4 (127.0.97.2:42081)
I20260620 02:39:04.241724 14337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:39:04.276528 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37319
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:04.632438 14668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:04.632810 14668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:04.632982 14668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:04.641402 14668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:04.641577 14668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:04.641778 14668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:39:04.652479 14668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:04.653855 14668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:04.655318 14668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:04.669480 14673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:04.669535 14674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:04.669512 14676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:04.670585 14668 server_base.cc:1034] running on GCE node
I20260620 02:39:04.671350 14668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:04.672477 14668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:04.673699 14668 hybrid_clock.cc:648] HybridClock initialized: now 1781923144673652 us; error 37 us; skew 500 ppm
I20260620 02:39:04.674069 14668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:04 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:04.677573 14668 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:39:04.679376 14668 webserver.cc:460] Webserver started at http://127.0.97.3:41097/ using document root <none> and password file <none>
I20260620 02:39:04.680207 14668 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:04.680399 14668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:04.680748 14668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:39:04.684021 14668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "2284be557f834092b535956fcf7b5eea"
format_stamp: "Formatted at 2026-06-20 02:39:04 on dist-test-slave-rpg6"
server_key: "3908758f42c130a51356a58324ebd0e2"
server_key_iv: "e49c8ff440ddae46e1fd56a92941278d"
server_key_version: "encryptionkey@0"
I20260620 02:39:04.684937 14668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2284be557f834092b535956fcf7b5eea"
format_stamp: "Formatted at 2026-06-20 02:39:04 on dist-test-slave-rpg6"
server_key: "3908758f42c130a51356a58324ebd0e2"
server_key_iv: "e49c8ff440ddae46e1fd56a92941278d"
server_key_version: "encryptionkey@0"
I20260620 02:39:04.688344 14528 heartbeater.cc:503] Master 127.0.97.62:37319 was elected leader, sending a full tablet report...
I20260620 02:39:04.691393 14668 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:39:04.696022 14683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:04.697597 14668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:39:04.697800 14668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "2284be557f834092b535956fcf7b5eea"
format_stamp: "Formatted at 2026-06-20 02:39:04 on dist-test-slave-rpg6"
server_key: "3908758f42c130a51356a58324ebd0e2"
server_key_iv: "e49c8ff440ddae46e1fd56a92941278d"
server_key_version: "encryptionkey@0"
I20260620 02:39:04.698055 14668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:04.714520 14668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:04.719262 14668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:04.719590 14668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:04.720912 14668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:04.722791 14668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:04.722910 14668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:04.723081 14668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:04.723166 14668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:04.761253 14668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44103
I20260620 02:39:04.761287 14795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44103 every 8 connection(s)
I20260620 02:39:04.762784 14668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:39:04.772807 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 14668
I20260620 02:39:04.773008 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:39:04.773524 388 external_mini_cluster.cc:1428] Setting key 13225fa568eb1a8f397c8fa90ec1fac8
Jun 20 02:39:04 dist-test-slave-rpg6 krb5kdc[14292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:04.785230 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:04.765768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53425 (local address 127.0.97.62:37319)
0620 02:39:04.766219 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:04.766223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:04.767316 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 02:39:04.769276 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:04.769282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:04.769388 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:04.769510 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:04.771198 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.772058 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.773911 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.774471 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.774626 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:04.778963 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:04.778969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:04.778972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:04.779010 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:04.781765 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:04.782625 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:04.782631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:04.782634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:04.782780 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:04.783259 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:04.783266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:04.783269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:04.783563 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:04.783736 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:04.784568 (+ 832us) server_negotiation.cc:299] Negotiation successful
0620 02:39:04.784837 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":221,"threads_started":1}
I20260620 02:39:04.786229 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:04.766351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:53425)
0620 02:39:04.767135 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:04.767177 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:04.768983 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:04.769765 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:04.769774 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:04.770328 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:04.770964 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:04.770982 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.772222 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.772233 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:04.773752 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:04.773766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.774733 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.774741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:04.775097 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:04.775989 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:04.776015 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:04.778707 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:04.781958 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:04.781969 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:04.781988 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:04.782483 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:04.782904 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:04.782909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:04.782913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:04.783129 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:04.783701 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:04.783711 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:04.784173 (+ 462us) client_negotiation.cc:769] Sending connection context
0620 02:39:04.784466 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 02:39:04.785043 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":325,"threads_started":1}
I20260620 02:39:04.787853 14796 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37319
I20260620 02:39:04.788237 14796 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:04.789112 14796 heartbeater.cc:511] Master 127.0.97.62:37319 requested a full tablet report, sending...
I20260620 02:39:04.790653 14337 ts_manager.cc:194] Registered new tserver with Master: 2284be557f834092b535956fcf7b5eea (127.0.97.3:44103)
I20260620 02:39:04.791538 14337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53425
I20260620 02:39:04.802605 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:39:04.821323 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:04.807906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.0.97.62:37319)
0620 02:39:04.808095 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:04.808100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:04.808550 (+ 450us) server_negotiation.cc:407] Connection header received
0620 02:39:04.808706 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:04.808711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:04.808781 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:04.808879 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:04.810309 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.811021 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.812460 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.812889 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.812998 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:04.815470 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:04.815475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:04.815478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:04.815517 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:04.818322 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:04.819021 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:04.819025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:04.819028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:04.819256 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:04.819662 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:04.819666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:04.819668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:04.820272 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:04.820484 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:04.820889 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 02:39:04.821079 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:39:04.826102 14337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:39:04.833783 14337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:39:04.876653 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:04.852355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.0.97.2:42081)
0620 02:39:04.866419 (+ 14064us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:04.866432 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:04.868038 (+ 1606us) server_negotiation.cc:407] Connection header received
0620 02:39:04.868164 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:04.868173 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:04.868418 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:04.868655 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:04.869360 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.870821 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.873826 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.875360 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.875518 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:04.875758 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:04.876116 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 02:39:04.876322 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13895,"thread_start_us":13786,"threads_started":1}
I20260620 02:39:04.876705 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:04.851632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42081 (local address 127.0.0.1:37340)
0620 02:39:04.867694 (+ 16062us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:04.867709 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:04.867891 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:04.868658 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:04.868665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:04.868693 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:04.869100 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:04.869117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.870965 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.870971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:04.873606 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:04.873625 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.875563 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.875569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:04.875795 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:04.875886 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 02:39:04.876032 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 02:39:04.876254 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15928,"mutex_wait_us":35,"thread_start_us":14942,"threads_started":1}
I20260620 02:39:04.877974 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:04.848731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.0.97.1:36939)
0620 02:39:04.868568 (+ 19837us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:04.868582 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:04.868646 (+ 64us) server_negotiation.cc:407] Connection header received
0620 02:39:04.868783 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:04.868795 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:04.869101 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:04.869948 (+ 847us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:04.870215 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.872029 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.875249 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.877001 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.877150 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:04.877404 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:04.877587 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 02:39:04.877725 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19566,"thread_start_us":20142,"threads_started":1}
I20260620 02:39:04.878043 14813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:04.848512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36939 (local address 127.0.0.1:54056)
0620 02:39:04.866936 (+ 18424us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:04.866964 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:04.867142 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:04.869382 (+ 2240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:04.869387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:04.869414 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:04.869770 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:04.869779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.872224 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.872232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:04.875067 (+ 2835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:04.875076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.877224 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.877230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:04.877408 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:04.877443 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:39:04.877549 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 02:39:04.877692 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18259,"mutex_wait_us":158,"thread_start_us":18898,"threads_started":1}
I20260620 02:39:04.879670 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:04.848739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.0.97.3:44103)
0620 02:39:04.869587 (+ 20848us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:04.869596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:04.869638 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:39:04.869734 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:04.869741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:04.869947 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:04.870144 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:04.870853 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.872036 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.875270 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.877228 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.878580 (+ 1352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:04.878809 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:04.879063 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 02:39:04.879258 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20675,"thread_start_us":20497,"threads_started":1}
I20260620 02:39:04.879689 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:04.848281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44103 (local address 127.0.0.1:39858)
0620 02:39:04.864540 (+ 16259us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:04.864600 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:04.864861 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:04.870202 (+ 5341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:04.870207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:04.870235 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:04.870641 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:04.870654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.872189 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.872197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:04.875026 (+ 2829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:04.875050 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:04.878666 (+ 3616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:04.878674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:04.878855 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:04.878889 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:39:04.878998 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 02:39:04.879250 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15895,"mutex_wait_us":187,"thread_start_us":16799,"threads_started":1}
I20260620 02:39:04.881404 14597 tablet_service.cc:1467] Processing CreateTablet for tablet dffa33ec75af4982abfe31a1379ed551 (DEFAULT_TABLE table=test-table [id=8e0adaca5d254813a595c2c3499e6ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:04.882246 14463 tablet_service.cc:1467] Processing CreateTablet for tablet dffa33ec75af4982abfe31a1379ed551 (DEFAULT_TABLE table=test-table [id=8e0adaca5d254813a595c2c3499e6ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:04.882910 14597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffa33ec75af4982abfe31a1379ed551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:04.883555 14463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffa33ec75af4982abfe31a1379ed551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:04.893321 14731 tablet_service.cc:1467] Processing CreateTablet for tablet dffa33ec75af4982abfe31a1379ed551 (DEFAULT_TABLE table=test-table [id=8e0adaca5d254813a595c2c3499e6ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:04.895056 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffa33ec75af4982abfe31a1379ed551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:04.896315 14815 tablet_bootstrap.cc:492] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: Bootstrap starting.
I20260620 02:39:04.899966 14815 tablet_bootstrap.cc:654] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:04.900806 14816 tablet_bootstrap.cc:492] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: Bootstrap starting.
I20260620 02:39:04.903096 14815 log.cc:824] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:04.905174 14816 tablet_bootstrap.cc:654] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:04.906831 14816 log.cc:824] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:04.907363 14815 tablet_bootstrap.cc:492] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: No bootstrap required, opened a new log
I20260620 02:39:04.907756 14815 ts_tablet_manager.cc:1397] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260620 02:39:04.910750 14819 tablet_bootstrap.cc:492] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: Bootstrap starting.
I20260620 02:39:04.914801 14819 tablet_bootstrap.cc:654] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:04.915103 14816 tablet_bootstrap.cc:492] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: No bootstrap required, opened a new log
I20260620 02:39:04.915493 14816 ts_tablet_manager.cc:1397] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260620 02:39:04.916435 14819 log.cc:824] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:04.919662 14819 tablet_bootstrap.cc:492] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: No bootstrap required, opened a new log
I20260620 02:39:04.920027 14819 ts_tablet_manager.cc:1397] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 02:39:04.919680 14815 raft_consensus.cc:348] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:04.920207 14815 raft_consensus.cc:374] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:04.920324 14815 raft_consensus.cc:729] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7175767fdecd49e3a35b46f248578b8f, State: Initialized, Role: FOLLOWER
I20260620 02:39:04.921005 14815 consensus_queue.cc:260] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:04.923022 14815 ts_tablet_manager.cc:1428] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260620 02:39:04.926045 14816 raft_consensus.cc:348] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:04.926584 14816 raft_consensus.cc:374] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:04.926785 14816 raft_consensus.cc:729] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42db81b4b7aa474ea0042bd0f2d87fd4, State: Initialized, Role: FOLLOWER
I20260620 02:39:04.927493 14816 consensus_queue.cc:260] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:04.928735 14662 heartbeater.cc:503] Master 127.0.97.62:37319 was elected leader, sending a full tablet report...
I20260620 02:39:04.929538 14816 ts_tablet_manager.cc:1428] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 02:39:04.930362 14819 raft_consensus.cc:348] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:04.930902 14819 raft_consensus.cc:374] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:04.931085 14819 raft_consensus.cc:729] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2284be557f834092b535956fcf7b5eea, State: Initialized, Role: FOLLOWER
I20260620 02:39:04.931881 14819 consensus_queue.cc:260] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:04.933069 14796 heartbeater.cc:503] Master 127.0.97.62:37319 was elected leader, sending a full tablet report...
I20260620 02:39:04.933736 14819 ts_tablet_manager.cc:1428] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260620 02:39:04.961043 14663 tablet.cc:2369] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:39:05.014662 14797 tablet.cc:2369] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:39:05.158329 14529 tablet.cc:2369] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:39:05.271306 14822 raft_consensus.cc:482] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:39:05.271773 14822 raft_consensus.cc:504] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:05.273672 14822 leader_election.cc:288] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2284be557f834092b535956fcf7b5eea (127.0.97.3:44103), 7175767fdecd49e3a35b46f248578b8f (127.0.97.1:36939)
I20260620 02:39:05.282734 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:05.274344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44513 (local address 127.0.97.1:36939)
0620 02:39:05.274591 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:05.274598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:05.275652 (+ 1054us) server_negotiation.cc:407] Connection header received
0620 02:39:05.276154 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:05.276160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:05.276251 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:05.276371 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:05.277232 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.278458 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.280669 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.281507 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.281631 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:05.281788 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:05.282271 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 02:39:05.282424 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:39:05.282769 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:05.274404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45341 (local address 127.0.97.3:44103)
0620 02:39:05.274617 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:05.274625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:05.275549 (+ 924us) server_negotiation.cc:407] Connection header received
0620 02:39:05.275791 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:05.275799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:05.275913 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:05.276035 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:05.277260 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.278062 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.280722 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.281562 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.281774 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:05.281882 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:05.282265 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 02:39:05.282446 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:39:05.282908 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:05.274391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44103 (local address 127.0.97.2:45341)
0620 02:39:05.275408 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:05.275426 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:05.275576 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:05.276175 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:05.276180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:05.276205 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:05.276833 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:05.276868 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.278262 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.278269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:05.280474 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:05.280490 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.281870 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.281875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:05.282077 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:05.282120 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:39:05.282236 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 02:39:05.282405 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":62,"thread_start_us":540,"threads_started":1}
I20260620 02:39:05.283200 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:05.274451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36939 (local address 127.0.97.2:44513)
0620 02:39:05.275542 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:05.275564 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:05.275713 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:05.276406 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:05.276409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:05.276429 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:05.276910 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:05.276922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.278645 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.278649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:05.280443 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:05.280461 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.281802 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.281808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:05.282062 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:05.282103 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:39:05.282236 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 02:39:05.282418 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":33,"thread_start_us":518,"threads_started":1}
I20260620 02:39:05.284009 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffa33ec75af4982abfe31a1379ed551" candidate_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2284be557f834092b535956fcf7b5eea" is_pre_election: true
I20260620 02:39:05.284365 14483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffa33ec75af4982abfe31a1379ed551" candidate_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7175767fdecd49e3a35b46f248578b8f" is_pre_election: true
I20260620 02:39:05.284546 14751 raft_consensus.cc:2454] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42db81b4b7aa474ea0042bd0f2d87fd4 in term 0.
I20260620 02:39:05.284879 14483 raft_consensus.cc:2454] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42db81b4b7aa474ea0042bd0f2d87fd4 in term 0.
I20260620 02:39:05.285257 14551 leader_election.cc:302] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2284be557f834092b535956fcf7b5eea, 42db81b4b7aa474ea0042bd0f2d87fd4; no voters:
I20260620 02:39:05.285727 14822 raft_consensus.cc:2781] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:39:05.285892 14822 raft_consensus.cc:482] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:39:05.286015 14822 raft_consensus.cc:3037] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:05.288540 14822 raft_consensus.cc:504] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:05.289268 14822 leader_election.cc:288] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [CANDIDATE]: Term 1 election: Requested vote from peers 2284be557f834092b535956fcf7b5eea (127.0.97.3:44103), 7175767fdecd49e3a35b46f248578b8f (127.0.97.1:36939)
I20260620 02:39:05.289922 14751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffa33ec75af4982abfe31a1379ed551" candidate_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2284be557f834092b535956fcf7b5eea"
I20260620 02:39:05.289896 14483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffa33ec75af4982abfe31a1379ed551" candidate_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7175767fdecd49e3a35b46f248578b8f"
I20260620 02:39:05.290153 14751 raft_consensus.cc:3037] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:05.290253 14483 raft_consensus.cc:3037] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:05.292682 14483 raft_consensus.cc:2454] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42db81b4b7aa474ea0042bd0f2d87fd4 in term 1.
I20260620 02:39:05.292677 14751 raft_consensus.cc:2454] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42db81b4b7aa474ea0042bd0f2d87fd4 in term 1.
I20260620 02:39:05.293228 14552 leader_election.cc:302] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42db81b4b7aa474ea0042bd0f2d87fd4, 7175767fdecd49e3a35b46f248578b8f; no voters:
I20260620 02:39:05.293568 14822 raft_consensus.cc:2781] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:39:05.294173 14822 raft_consensus.cc:686] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 42db81b4b7aa474ea0042bd0f2d87fd4, State: Running, Role: LEADER
I20260620 02:39:05.295090 14822 consensus_queue.cc:237] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:05.300591 14336 catalog_manager.cc:5495] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42db81b4b7aa474ea0042bd0f2d87fd4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } health_report { overall_health: HEALTHY } } }
I20260620 02:39:05.325098 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:05.318835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.0.97.2:42081)
0620 02:39:05.319120 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:05.319126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:05.319151 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:39:05.319506 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:05.319511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:05.319619 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:05.319757 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:39:05.320204 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.321028 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.322664 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.323105 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.323210 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:05.323738 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:39:05.324382 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:39:05.324505 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:05.324649 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 02:39:05.324742 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 02:39:05.334738 14483 raft_consensus.cc:1261] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f [term 1 FOLLOWER]: Refusing update from remote peer 42db81b4b7aa474ea0042bd0f2d87fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:39:05.334738 14751 raft_consensus.cc:1261] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea [term 1 FOLLOWER]: Refusing update from remote peer 42db81b4b7aa474ea0042bd0f2d87fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:39:05.336037 14826 consensus_queue.cc:1035] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:39:05.336535 14822 consensus_queue.cc:1035] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:39:05.356572 14830 mvcc.cc:205] Tried to move back new op lower bound from 7298757203279196160 to 7298757203133665280. Current Snapshot: MvccSnapshot[applied={T|T < 7298757203279196160}]
I20260620 02:39:05.369344 14337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:39:05.369712 14337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:39:05.374475 14337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Sending DeleteTablet for 3 replicas of tablet dffa33ec75af4982abfe31a1379ed551
I20260620 02:39:05.375924 14731 tablet_service.cc:1514] Processing DeleteTablet for tablet dffa33ec75af4982abfe31a1379ed551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39858
I20260620 02:39:05.375929 14463 tablet_service.cc:1514] Processing DeleteTablet for tablet dffa33ec75af4982abfe31a1379ed551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54056
I20260620 02:39:05.376770 14597 tablet_service.cc:1514] Processing DeleteTablet for tablet dffa33ec75af4982abfe31a1379ed551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37340
I20260620 02:39:05.377172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 14400
I20260620 02:39:05.378470 14839 tablet_replica.cc:330] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: stopping tablet replica
I20260620 02:39:05.379271 14839 raft_consensus.cc:2229] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:39:05.380470 14840 tablet_replica.cc:330] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: stopping tablet replica
I20260620 02:39:05.380519 14839 raft_consensus.cc:2258] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:05.381095 14840 raft_consensus.cc:2229] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:39:05.381472 14841 tablet_replica.cc:330] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: stopping tablet replica
I20260620 02:39:05.381937 14840 raft_consensus.cc:2258] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:05.381935 14841 raft_consensus.cc:2229] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:39:05.382658 14841 raft_consensus.cc:2258] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:05.384686 14839 ts_tablet_manager.cc:1905] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:39:05.384783 14840 ts_tablet_manager.cc:1905] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:39:05.392735 14839 ts_tablet_manager.cc:1918] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:05.392836 14840 ts_tablet_manager.cc:1918] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:05.393028 14840 log.cc:1196] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/dffa33ec75af4982abfe31a1379ed551
I20260620 02:39:05.393616 14840 ts_tablet_manager.cc:1939] T dffa33ec75af4982abfe31a1379ed551 P 7175767fdecd49e3a35b46f248578b8f: Deleting consensus metadata
I20260620 02:39:05.394815 14841 ts_tablet_manager.cc:1905] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:39:05 dist-test-slave-rpg6 krb5kdc[14292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:39:05.396097 14323 catalog_manager.cc:4841] TS 7175767fdecd49e3a35b46f248578b8f (127.0.97.1:36939): tablet dffa33ec75af4982abfe31a1379ed551 (table test-table [id=8e0adaca5d254813a595c2c3499e6ef7]) successfully deleted
I20260620 02:39:05.396678 14839 log.cc:1196] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/dffa33ec75af4982abfe31a1379ed551
I20260620 02:39:05.397297 14839 ts_tablet_manager.cc:1939] T dffa33ec75af4982abfe31a1379ed551 P 2284be557f834092b535956fcf7b5eea: Deleting consensus metadata
I20260620 02:39:05.399474 14322 catalog_manager.cc:4841] TS 2284be557f834092b535956fcf7b5eea (127.0.97.3:44103): tablet dffa33ec75af4982abfe31a1379ed551 (table test-table [id=8e0adaca5d254813a595c2c3499e6ef7]) successfully deleted
I20260620 02:39:05.403371 14841 ts_tablet_manager.cc:1918] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:05.403614 14841 log.cc:1196] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/dffa33ec75af4982abfe31a1379ed551
I20260620 02:39:05.404227 14841 ts_tablet_manager.cc:1939] T dffa33ec75af4982abfe31a1379ed551 P 42db81b4b7aa474ea0042bd0f2d87fd4: Deleting consensus metadata
I20260620 02:39:05.406216 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:05.378772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.0.97.1:36939)
0620 02:39:05.379029 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:05.379039 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:05.383512 (+ 4473us) server_negotiation.cc:407] Connection header received
0620 02:39:05.383877 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:05.383883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:05.383974 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:05.384058 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:05.386792 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.388007 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.390220 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.390910 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.391026 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:05.397575 (+ 6549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:05.397594 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:05.397615 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:05.397672 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:05.401896 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:05.403051 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:05.403064 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:05.403070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:05.403259 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:05.403767 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:05.403773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:05.403776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:05.404368 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:05.404707 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:05.405439 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 02:39:05.405802 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:39:05.406342 14321 catalog_manager.cc:4841] TS 42db81b4b7aa474ea0042bd0f2d87fd4 (127.0.97.2:42081): tablet dffa33ec75af4982abfe31a1379ed551 (table test-table [id=8e0adaca5d254813a595c2c3499e6ef7]) successfully deleted
I20260620 02:39:05.407038 14523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:05.498036 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 14400
I20260620 02:39:05.522336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 14534
Jun 20 02:39:05 dist-test-slave-rpg6 krb5kdc[14292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:39:05.546738 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:05.523171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.0.97.2:42081)
0620 02:39:05.523385 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:05.523391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:05.523414 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:39:05.523494 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:05.523499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:05.523579 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:05.523693 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:05.525313 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.526667 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.528472 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.529169 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.529332 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:05.538434 (+ 9102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:05.538459 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:05.538481 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:05.538544 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:05.543125 (+ 4581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:05.544291 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:05.544302 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:05.544306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:05.544483 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:05.544914 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:05.544919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:05.544923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:05.545434 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:05.545759 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:05.546049 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 02:39:05.546302 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:39:05.547513 14657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:05.666195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 14534
I20260620 02:39:05.686096 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 14668
Jun 20 02:39:05 dist-test-slave-rpg6 krb5kdc[14292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:39:05.702809 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:05.686918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.0.97.3:44103)
0620 02:39:05.687084 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:05.687088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:05.687107 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:39:05.687252 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:05.687256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:05.687325 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:05.687425 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:05.689258 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.690004 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.691474 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:05.692011 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:05.692128 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:05.696824 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:05.696840 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:05.696856 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:05.696897 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:05.699672 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:05.700538 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:05.700547 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:05.700549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:05.700714 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:05.701179 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:05.701184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:05.701186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:05.701584 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:05.701820 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:05.702296 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 02:39:05.702527 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:39:05.703603 14791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:05.796084 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 14668
I20260620 02:39:05.813665 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 14305
I20260620 02:39:05.814965 14367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:05.932533 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 14305
I20260620 02:39:05.954624 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37319
--webserver_interface=127.0.97.62
--webserver_port=45153
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:06.329078 14847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:06.329424 14847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:06.329588 14847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:06.337316 14847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:39:06.337450 14847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:06.337502 14847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:06.337553 14847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:39:06.337656 14847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:39:06.348270 14847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:06.349735 14847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:06.351130 14847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:06.376034 14854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:06.376431 14853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:06.376875 14856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:06.377349 14847 server_base.cc:1034] running on GCE node
I20260620 02:39:06.378289 14847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:06.380153 14847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:06.381451 14847 hybrid_clock.cc:648] HybridClock initialized: now 1781923146381394 us; error 50 us; skew 500 ppm
I20260620 02:39:06.381939 14847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:06 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:06.393249 14847 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:39:06.395463 14847 webserver.cc:460] Webserver started at http://127.0.97.62:45153/ using document root <none> and password file <none>
I20260620 02:39:06.396446 14847 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:06.396651 14847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:06.404500 14847 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 02:39:06.408692 14863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:06.410494 14847 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:39:06.410737 14847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3"
format_stamp: "Formatted at 2026-06-20 02:39:02 on dist-test-slave-rpg6"
server_key: "3bf9fce644a60c2a9b094ee90c375fd3"
server_key_iv: "fdaba8ca6946450cc5e27e42abfeddbd"
server_key_version: "encryptionkey@0"
I20260620 02:39:06.412402 14847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:06.434487 14847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:06.440227 14847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:06.440603 14847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:06.464290 14847 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37319
I20260620 02:39:06.464308 14914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37319 every 8 connection(s)
I20260620 02:39:06.466163 14847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:39:06.472832 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 14847
I20260620 02:39:06.473845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36939
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37319
--webserver_port=35623
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:39:06.474695 14915 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:39:06.480386 14915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Bootstrap starting.
I20260620 02:39:06.501120 14915 log.cc:824] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:06.515575 14915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:39:06.516294 14915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Bootstrap complete.
I20260620 02:39:06.526710 14915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:06.527763 14915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ebf31f3bb34d23a0afec81d5eb1fc3, State: Initialized, Role: FOLLOWER
I20260620 02:39:06.528563 14915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:06.528860 14915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:39:06.529016 14915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:39:06.529212 14915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:39:06.533713 14915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:06.534461 14915 leader_election.cc:302] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52ebf31f3bb34d23a0afec81d5eb1fc3; no voters:
I20260620 02:39:06.535183 14915 leader_election.cc:288] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:39:06.535328 14919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:39:06.536816 14919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 2 LEADER]: Becoming Leader. State: Replica: 52ebf31f3bb34d23a0afec81d5eb1fc3, State: Running, Role: LEADER
I20260620 02:39:06.537562 14919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:06.537950 14915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:39:06.540315 14921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ebf31f3bb34d23a0afec81d5eb1fc3. Latest consensus state: current_term: 2 leader_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } } }
I20260620 02:39:06.540747 14921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:06.541460 14920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } } }
I20260620 02:39:06.541816 14920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:06.554719 14926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:39:06.570451 14926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:39:06.573120 14926 catalog_manager.cc:1259] Loaded cluster ID: 8dd9974dad7b466989fa5033525b38ee
I20260620 02:39:06.573369 14926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:39:06.579427 14926 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 02:39:06.602764 14926 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 02:39:06.611332 14926 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:39:06.612918 14926 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Loaded TSK: 0
I20260620 02:39:06.614425 14926 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:39:06.889510 14917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:06.889840 14917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:06.890007 14917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:06.898135 14917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:06.898343 14917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:06.898543 14917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:39:06.909885 14917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35623
--tserver_master_addrs=127.0.97.62:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:06.911355 14917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:06.913546 14917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:06.928109 14942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:06.928087 14945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:06.928279 14943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:06.929646 14917 server_base.cc:1034] running on GCE node
I20260620 02:39:06.930361 14917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:06.931571 14917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:06.932812 14917 hybrid_clock.cc:648] HybridClock initialized: now 1781923146932694 us; error 113 us; skew 500 ppm
I20260620 02:39:06.933192 14917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:06 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:06.936657 14917 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:39:06.938424 14917 webserver.cc:460] Webserver started at http://127.0.97.1:35623/ using document root <none> and password file <none>
I20260620 02:39:06.939170 14917 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:06.939342 14917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:06.945508 14917 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 02:39:06.948621 14952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:06.950197 14917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:39:06.950403 14917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "7175767fdecd49e3a35b46f248578b8f"
format_stamp: "Formatted at 2026-06-20 02:39:03 on dist-test-slave-rpg6"
server_key: "d6a9aa1b087ff9bff18bb84d91aebd45"
server_key_iv: "048e748175302dad63494cf3fb5bb21c"
server_key_version: "encryptionkey@0"
I20260620 02:39:06.951545 14917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:06.973229 14917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:06.978305 14917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:06.978632 14917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:06.979755 14917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:06.981170 14917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:06.981271 14917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:06.981439 14917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:06.981537 14917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:07.019042 14917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36939
I20260620 02:39:07.019202 15064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36939 every 8 connection(s)
I20260620 02:39:07.020665 14917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:39:07.023219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 14917
I20260620 02:39:07.024082 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42081
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37319
--webserver_port=39477
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:39:07 dist-test-slave-rpg6 krb5kdc[14292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:07.049412 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:07.024404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:51587)
0620 02:39:07.025416 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:07.025472 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:07.027407 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:07.029216 (+ 1809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:07.029224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:07.029827 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:07.030432 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:07.030454 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:07.032422 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:07.032431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:07.034789 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:07.034813 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:07.036185 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:07.036195 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:07.036532 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:07.037400 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:07.037444 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:07.040571 (+ 3127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:07.044651 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:07.044660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:07.044674 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:07.045143 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:07.045668 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:07.045674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:07.045677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:07.045870 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:07.046691 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:07.046700 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:07.047051 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 02:39:07.047340 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 02:39:07.048011 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":462,"threads_started":1}
I20260620 02:39:07.049412 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:07.024105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51587 (local address 127.0.97.62:37319)
0620 02:39:07.024898 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:07.024918 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:07.025649 (+ 731us) server_negotiation.cc:407] Connection header received
0620 02:39:07.028118 (+ 2469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:07.028139 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:07.028656 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:07.029072 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:07.030722 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:07.032206 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:07.035014 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:07.035804 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:07.036075 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:07.040991 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:07.041011 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:07.041030 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:07.041075 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:07.044414 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:07.045320 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:07.045328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:07.045333 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:07.045517 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:07.046003 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:07.046014 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:07.046017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:07.046518 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:07.046734 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:07.047546 (+ 812us) server_negotiation.cc:299] Negotiation successful
0620 02:39:07.048011 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":374,"threads_started":1}
I20260620 02:39:07.052574 15065 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37319
I20260620 02:39:07.052922 15065 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:07.053704 15065 heartbeater.cc:511] Master 127.0.97.62:37319 requested a full tablet report, sending...
I20260620 02:39:07.056279 14880 ts_manager.cc:194] Registered new tserver with Master: 7175767fdecd49e3a35b46f248578b8f (127.0.97.1:36939)
I20260620 02:39:07.057854 14880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51587
W20260620 02:39:07.380445 15069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:07.380775 15069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:07.380873 15069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:07.388757 15069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:07.388901 15069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:07.389019 15069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:39:07.399494 15069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39477
--tserver_master_addrs=127.0.97.62:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:07.400931 15069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:07.402277 15069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:07.416726 15074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:07.418350 15075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:07.418592 15077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:07.419564 15069 server_base.cc:1034] running on GCE node
I20260620 02:39:07.420457 15069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:07.421491 15069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:07.422782 15069 hybrid_clock.cc:648] HybridClock initialized: now 1781923147422695 us; error 37 us; skew 500 ppm
I20260620 02:39:07.423264 15069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:07 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:07.427011 15069 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:39:07.428818 15069 webserver.cc:460] Webserver started at http://127.0.97.2:39477/ using document root <none> and password file <none>
I20260620 02:39:07.429608 15069 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:07.429802 15069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:07.435688 15069 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:39:07.438772 15084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:07.440356 15069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:39:07.440593 15069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "42db81b4b7aa474ea0042bd0f2d87fd4"
format_stamp: "Formatted at 2026-06-20 02:39:04 on dist-test-slave-rpg6"
server_key: "ecfffbb8962cdcc53e3b78c854ade989"
server_key_iv: "6d79f7d347c8cc6833d2cf169c7ca4fa"
server_key_version: "encryptionkey@0"
I20260620 02:39:07.441741 15069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:07.478449 15069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:07.483294 15069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:07.483603 15069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:07.484745 15069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:07.486225 15069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:07.486354 15069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:07.486496 15069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:07.486588 15069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:07.523741 15069 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42081
I20260620 02:39:07.523801 15196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42081 every 8 connection(s)
I20260620 02:39:07.525336 15069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:39:07.533457 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 15069
I20260620 02:39:07.534300 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44103
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37319
--webserver_port=41097
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:39:07 dist-test-slave-rpg6 krb5kdc[14292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:07.553608 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:07.528407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53065 (local address 127.0.97.62:37319)
0620 02:39:07.528640 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:07.528645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:07.529920 (+ 1275us) server_negotiation.cc:407] Connection header received
0620 02:39:07.532056 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:07.532061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:07.532154 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:07.532258 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:07.534844 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:07.535629 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:07.538418 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:07.538873 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:07.539166 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:07.547179 (+ 8013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:07.547188 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:07.547198 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:07.547241 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:07.550074 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:07.551036 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:07.551040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:07.551043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:07.551187 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:07.551689 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:07.551693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:07.551695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:07.552051 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:07.552226 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:07.552977 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 02:39:07.553234 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:39:07.554658 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:07.528826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:53065)
0620 02:39:07.529709 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:07.529755 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:07.531658 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:07.532534 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:07.532544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:07.533482 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:07.534466 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:07.534501 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:07.535885 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:07.535894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:07.538215 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:07.538239 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:07.539268 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:07.539277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:07.539539 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:07.544173 (+ 4634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:07.544209 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:07.546802 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:07.550316 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:07.550327 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:07.550348 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:07.550857 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:07.551313 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:07.551318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:07.551322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:07.551551 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:07.552188 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:07.552198 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:07.552595 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 02:39:07.552884 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 02:39:07.553470 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":333,"threads_started":1}
I20260620 02:39:07.556622 15197 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37319
I20260620 02:39:07.556978 15197 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:07.557780 15197 heartbeater.cc:511] Master 127.0.97.62:37319 requested a full tablet report, sending...
I20260620 02:39:07.559520 14880 ts_manager.cc:194] Registered new tserver with Master: 42db81b4b7aa474ea0042bd0f2d87fd4 (127.0.97.2:42081)
I20260620 02:39:07.560719 14880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53065
W20260620 02:39:07.903422 15200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:07.903738 15200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:07.903924 15200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:07.911614 15200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:07.911743 15200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:07.911935 15200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:39:07.922163 15200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41097
--tserver_master_addrs=127.0.97.62:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:07.923553 15200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:07.925047 15200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:07.939973 15205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:07.939965 15208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:07.940095 15206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:07.940726 15200 server_base.cc:1034] running on GCE node
I20260620 02:39:07.941460 15200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:07.942490 15200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:07.943732 15200 hybrid_clock.cc:648] HybridClock initialized: now 1781923147943653 us; error 58 us; skew 500 ppm
I20260620 02:39:07.944092 15200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:07 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:07.947640 15200 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:39:07.949361 15200 webserver.cc:460] Webserver started at http://127.0.97.3:41097/ using document root <none> and password file <none>
I20260620 02:39:07.950116 15200 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:07.950296 15200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:07.956642 15200 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:39:07.960100 15215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:07.962116 15200 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:39:07.962345 15200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "2284be557f834092b535956fcf7b5eea"
format_stamp: "Formatted at 2026-06-20 02:39:04 on dist-test-slave-rpg6"
server_key: "3908758f42c130a51356a58324ebd0e2"
server_key_iv: "e49c8ff440ddae46e1fd56a92941278d"
server_key_version: "encryptionkey@0"
I20260620 02:39:07.963438 15200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:07.981799 15200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:07.986775 15200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:07.987152 15200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:07.988366 15200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:07.989886 15200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:07.989988 15200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:07.990156 15200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:07.990252 15200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:08.029040 15200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44103
I20260620 02:39:08.029142 15327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44103 every 8 connection(s)
I20260620 02:39:08.030584 15200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:39:08.032629 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 15200
Jun 20 02:39:08 dist-test-slave-rpg6 krb5kdc[14292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:08.052481 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.033712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.0.97.62:37319)
0620 02:39:08.033971 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.033981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.034390 (+ 409us) server_negotiation.cc:407] Connection header received
0620 02:39:08.034611 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.034620 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.034734 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.034867 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:08.037144 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.038347 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.040569 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.041062 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.041174 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.045127 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:08.045134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:08.045139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:08.045183 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:08.049170 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:08.050000 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:08.050013 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:08.050015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:08.050151 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:08.050660 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:08.050669 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:08.050671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:08.051008 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:08.051383 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.051738 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.052059 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":78}
I20260620 02:39:08.057830 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.034251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60111 (local address 127.0.97.62:37319)
0620 02:39:08.035309 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.035316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.035977 (+ 661us) server_negotiation.cc:407] Connection header received
0620 02:39:08.038532 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.038538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.038640 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.038775 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:08.040636 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.041577 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.043526 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.044575 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.044737 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.050449 (+ 5712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:08.050456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:08.050462 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:08.050500 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:08.053915 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:08.054909 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:08.054914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:08.054918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:08.055131 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:08.055883 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:08.055889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:08.055892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:08.056206 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:08.056390 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.057150 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.057396 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":849,"threads_started":1}
I20260620 02:39:08.058832 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.034531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:60111)
0620 02:39:08.035765 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.035813 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:08.038087 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:08.039015 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:08.039024 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:08.039677 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:08.040404 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.040426 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.041800 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.041811 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.043311 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.043325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.044890 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.044898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.045404 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.046478 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:08.046521 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:08.050197 (+ 3676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:08.054140 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:08.054152 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:08.054175 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:08.054768 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:08.055388 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:08.055393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:08.055396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:08.055695 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:08.056373 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:08.056381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:08.056765 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 02:39:08.057051 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 02:39:08.057659 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":704,"threads_started":1}
I20260620 02:39:08.060626 15328 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37319
I20260620 02:39:08.061475 15328 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:08.061857 15065 heartbeater.cc:503] Master 127.0.97.62:37319 was elected leader, sending a full tablet report...
I20260620 02:39:08.062913 15328 heartbeater.cc:511] Master 127.0.97.62:37319 requested a full tablet report, sending...
I20260620 02:39:08.065192 14880 ts_manager.cc:194] Registered new tserver with Master: 2284be557f834092b535956fcf7b5eea (127.0.97.3:44103)
I20260620 02:39:08.067144 14880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60111
I20260620 02:39:08.068928 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:39:08.077090 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.069946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54194 (local address 127.0.97.62:37319)
0620 02:39:08.070170 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.070179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.070694 (+ 515us) server_negotiation.cc:407] Connection header received
0620 02:39:08.070851 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.070855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.071075 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.071206 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:39:08.072111 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.072837 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.074545 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.074983 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.075094 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.075665 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:39:08.076408 (+ 743us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:39:08.076524 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.076688 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.076765 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:39:08.078320 14880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:39:08.086032 14880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:39:08.124891 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.101487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42081 (local address 127.0.0.1:37362)
0620 02:39:08.114997 (+ 13510us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.115020 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:08.115172 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:08.116603 (+ 1431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:08.116612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:08.116640 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:08.117049 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.117065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.119097 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.119103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.121840 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.121853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.123396 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.123402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.123602 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.123660 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 02:39:08.123798 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 02:39:08.124406 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13332,"mutex_wait_us":59,"thread_start_us":13210,"threads_started":1}
I20260620 02:39:08.125617 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.100957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44103 (local address 127.0.0.1:39874)
0620 02:39:08.116223 (+ 15266us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.116254 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:08.116428 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:08.117813 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:08.117818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:08.117842 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:08.118209 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.118219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.120349 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.120354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.122749 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.122801 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.124901 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.124907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.125103 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.125133 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:39:08.125221 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 02:39:08.125355 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15090,"mutex_wait_us":1062,"thread_start_us":13916,"threads_started":1}
I20260620 02:39:08.126017 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.101534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.0.97.2:42081)
0620 02:39:08.111500 (+ 9966us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.111516 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.115334 (+ 3818us) server_negotiation.cc:407] Connection header received
0620 02:39:08.115489 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.115500 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.115813 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.116047 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:08.117294 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.118952 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.122025 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.123179 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.123318 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.123539 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.125473 (+ 1934us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.125673 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9715,"thread_start_us":9533,"threads_started":1}
I20260620 02:39:08.126376 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.101612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.0.97.3:44103)
0620 02:39:08.117111 (+ 15499us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.117122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.117159 (+ 37us) server_negotiation.cc:407] Connection header received
0620 02:39:08.117263 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.117274 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.117556 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.117814 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:08.118427 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.120154 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.123017 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.124417 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.124556 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.124792 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.125877 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.126023 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15304,"thread_start_us":15354,"threads_started":1}
I20260620 02:39:08.126912 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.100957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36939 (local address 127.0.0.1:54080)
0620 02:39:08.113884 (+ 12927us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.113948 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:08.114222 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:08.118354 (+ 4132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:08.118359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:08.118386 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:08.118768 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.118781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.120824 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.120829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.122985 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.123003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.126109 (+ 3106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.126116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.126313 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.126345 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:39:08.126451 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 02:39:08.126580 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12538,"thread_start_us":15289,"threads_started":1}
I20260620 02:39:08.128135 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.101035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.0.97.1:36939)
0620 02:39:08.117798 (+ 16763us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.117806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.117834 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:39:08.117926 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.117933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.118137 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.118336 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:08.118984 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.120655 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.123143 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.125595 (+ 2452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.125739 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.125955 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.126682 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.126876 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16550,"thread_start_us":16342,"threads_started":1}
I20260620 02:39:08.130167 15132 tablet_service.cc:1467] Processing CreateTablet for tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9 (DEFAULT_TABLE table=test-table [id=fa337788e4b1485383717cb8247a04c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:08.130235 15263 tablet_service.cc:1467] Processing CreateTablet for tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9 (DEFAULT_TABLE table=test-table [id=fa337788e4b1485383717cb8247a04c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:08.131471 15132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:08.131620 15263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:08.131959 15000 tablet_service.cc:1467] Processing CreateTablet for tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9 (DEFAULT_TABLE table=test-table [id=fa337788e4b1485383717cb8247a04c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:08.133500 15000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:08.145903 15344 tablet_bootstrap.cc:492] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: Bootstrap starting.
I20260620 02:39:08.148219 15345 tablet_bootstrap.cc:492] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: Bootstrap starting.
I20260620 02:39:08.149917 15346 tablet_bootstrap.cc:492] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: Bootstrap starting.
I20260620 02:39:08.151571 15344 tablet_bootstrap.cc:654] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:08.151865 15345 tablet_bootstrap.cc:654] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:08.152872 15344 log.cc:824] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:08.153190 15345 log.cc:824] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:08.154322 15346 tablet_bootstrap.cc:654] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:08.155773 15346 log.cc:824] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:08.156250 15344 tablet_bootstrap.cc:492] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: No bootstrap required, opened a new log
I20260620 02:39:08.156574 15344 ts_tablet_manager.cc:1397] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 02:39:08.157357 15345 tablet_bootstrap.cc:492] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: No bootstrap required, opened a new log
I20260620 02:39:08.157629 15345 ts_tablet_manager.cc:1397] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 02:39:08.164407 15344 raft_consensus.cc:348] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:08.164908 15344 raft_consensus.cc:374] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:08.165011 15344 raft_consensus.cc:729] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42db81b4b7aa474ea0042bd0f2d87fd4, State: Initialized, Role: FOLLOWER
I20260620 02:39:08.165858 15344 consensus_queue.cc:260] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:08.167001 15346 tablet_bootstrap.cc:492] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: No bootstrap required, opened a new log
I20260620 02:39:08.167138 15197 heartbeater.cc:503] Master 127.0.97.62:37319 was elected leader, sending a full tablet report...
I20260620 02:39:08.167470 15346 ts_tablet_manager.cc:1397] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260620 02:39:08.168283 15344 ts_tablet_manager.cc:1428] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 02:39:08.171355 15345 raft_consensus.cc:348] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:08.171950 15345 raft_consensus.cc:374] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:08.172060 15345 raft_consensus.cc:729] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2284be557f834092b535956fcf7b5eea, State: Initialized, Role: FOLLOWER
I20260620 02:39:08.172773 15345 consensus_queue.cc:260] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:08.174782 15328 heartbeater.cc:503] Master 127.0.97.62:37319 was elected leader, sending a full tablet report...
I20260620 02:39:08.175469 15345 ts_tablet_manager.cc:1428] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260620 02:39:08.178606 15346 raft_consensus.cc:348] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:08.179049 15346 raft_consensus.cc:374] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:08.179150 15346 raft_consensus.cc:729] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7175767fdecd49e3a35b46f248578b8f, State: Initialized, Role: FOLLOWER
I20260620 02:39:08.179894 15346 consensus_queue.cc:260] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:08.182013 15346 ts_tablet_manager.cc:1428] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
W20260620 02:39:08.273445 15066 tablet.cc:2369] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:39:08.277742 15198 tablet.cc:2369] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:39:08.282601 15329 tablet.cc:2369] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:39:08.370508 15350 raft_consensus.cc:482] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:39:08.370891 15350 raft_consensus.cc:504] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:08.372794 15350 leader_election.cc:288] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7175767fdecd49e3a35b46f248578b8f (127.0.97.1:36939), 2284be557f834092b535956fcf7b5eea (127.0.97.3:44103)
I20260620 02:39:08.380864 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.373273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37539 (local address 127.0.97.1:36939)
0620 02:39:08.373524 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.373533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.375212 (+ 1679us) server_negotiation.cc:407] Connection header received
0620 02:39:08.375494 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.375498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.375566 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.375673 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:08.376309 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.377056 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.378980 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.379767 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.380027 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.380143 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.380445 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.380577 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:39:08.380957 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.373274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36939 (local address 127.0.97.2:37539)
0620 02:39:08.375067 (+ 1793us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.375088 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:08.375319 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:08.375702 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:08.375707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:08.375733 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:08.376153 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.376162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.377199 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.377206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.378814 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.378827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.380095 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.380101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.380258 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.380283 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:39:08.380416 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 02:39:08.380572 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1598,"thread_start_us":1224,"threads_started":1}
I20260620 02:39:08.381227 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.373454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60471 (local address 127.0.97.3:44103)
0620 02:39:08.373680 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.373686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.375315 (+ 1629us) server_negotiation.cc:407] Connection header received
0620 02:39:08.375544 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.375550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.375618 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.375740 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:08.376501 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.377252 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.379190 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.380142 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.380314 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.380438 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.380759 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.380903 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:39:08.381834 15354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.373329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44103 (local address 127.0.97.2:60471)
0620 02:39:08.375195 (+ 1866us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.375212 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:08.375348 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:08.375752 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:08.375757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:08.375787 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:08.376316 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.376326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.377415 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.377432 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.378991 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:08.379004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.380386 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.380393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:08.380579 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.380603 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:39:08.380733 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 02:39:08.380873 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1712,"mutex_wait_us":4,"spinlock_wait_cycles":24448,"thread_start_us":1420,"threads_started":1}
I20260620 02:39:08.382308 15020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe96a9fdd4f4f7fac36e0a7b015c7c9" candidate_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7175767fdecd49e3a35b46f248578b8f" is_pre_election: true
I20260620 02:39:08.383014 15020 raft_consensus.cc:2454] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42db81b4b7aa474ea0042bd0f2d87fd4 in term 0.
I20260620 02:39:08.383049 15283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe96a9fdd4f4f7fac36e0a7b015c7c9" candidate_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2284be557f834092b535956fcf7b5eea" is_pre_election: true
I20260620 02:39:08.383646 15283 raft_consensus.cc:2454] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42db81b4b7aa474ea0042bd0f2d87fd4 in term 0.
I20260620 02:39:08.383875 15087 leader_election.cc:302] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42db81b4b7aa474ea0042bd0f2d87fd4, 7175767fdecd49e3a35b46f248578b8f; no voters:
I20260620 02:39:08.384402 15350 raft_consensus.cc:2781] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:39:08.384603 15350 raft_consensus.cc:482] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:39:08.384739 15350 raft_consensus.cc:3037] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:08.387269 15350 raft_consensus.cc:504] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:08.388227 15350 leader_election.cc:288] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [CANDIDATE]: Term 1 election: Requested vote from peers 7175767fdecd49e3a35b46f248578b8f (127.0.97.1:36939), 2284be557f834092b535956fcf7b5eea (127.0.97.3:44103)
I20260620 02:39:08.388873 15020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe96a9fdd4f4f7fac36e0a7b015c7c9" candidate_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7175767fdecd49e3a35b46f248578b8f"
I20260620 02:39:08.388873 15283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe96a9fdd4f4f7fac36e0a7b015c7c9" candidate_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2284be557f834092b535956fcf7b5eea"
I20260620 02:39:08.389124 15020 raft_consensus.cc:3037] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:08.389129 15283 raft_consensus.cc:3037] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:08.391441 15020 raft_consensus.cc:2454] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42db81b4b7aa474ea0042bd0f2d87fd4 in term 1.
I20260620 02:39:08.391440 15283 raft_consensus.cc:2454] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42db81b4b7aa474ea0042bd0f2d87fd4 in term 1.
I20260620 02:39:08.392027 15086 leader_election.cc:302] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2284be557f834092b535956fcf7b5eea, 42db81b4b7aa474ea0042bd0f2d87fd4; no voters:
I20260620 02:39:08.392441 15350 raft_consensus.cc:2781] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:39:08.392939 15350 raft_consensus.cc:686] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 42db81b4b7aa474ea0042bd0f2d87fd4, State: Running, Role: LEADER
I20260620 02:39:08.393451 15350 consensus_queue.cc:237] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } } peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } }
I20260620 02:39:08.398665 14880 catalog_manager.cc:5495] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42db81b4b7aa474ea0042bd0f2d87fd4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42db81b4b7aa474ea0042bd0f2d87fd4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42081 } health_report { overall_health: HEALTHY } } }
I20260620 02:39:08.428443 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.422687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.0.97.2:42081)
0620 02:39:08.422954 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.422959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.422977 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:39:08.423067 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.423070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.423167 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.423306 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:39:08.423681 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.424604 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.426062 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.426510 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.426617 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.427131 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:39:08.427693 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:39:08.427807 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.428014 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.428105 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 02:39:08.438017 15283 raft_consensus.cc:1261] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea [term 1 FOLLOWER]: Refusing update from remote peer 42db81b4b7aa474ea0042bd0f2d87fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:39:08.438016 15020 raft_consensus.cc:1261] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f [term 1 FOLLOWER]: Refusing update from remote peer 42db81b4b7aa474ea0042bd0f2d87fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:39:08.439198 15355 consensus_queue.cc:1035] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7175767fdecd49e3a35b46f248578b8f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:39:08.439607 15350 consensus_queue.cc:1035] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2284be557f834092b535956fcf7b5eea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:39:08.456626 15358 mvcc.cc:205] Tried to move back new op lower bound from 7298757215989841920 to 7298757215823650816. Current Snapshot: MvccSnapshot[applied={T|T < 7298757215989841920}]
I20260620 02:39:08.472173 14879 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:39:08.472597 14879 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:39:08.476815 14879 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Sending DeleteTablet for 3 replicas of tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9
I20260620 02:39:08.478304 15132 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37362
I20260620 02:39:08.478300 15000 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54080
I20260620 02:39:08.480005 15263 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39874
I20260620 02:39:08.482501 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 14917
I20260620 02:39:08.487114 15367 tablet_replica.cc:330] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: stopping tablet replica
I20260620 02:39:08.487643 15367 raft_consensus.cc:2229] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:39:08.488232 15367 raft_consensus.cc:2258] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:08.489113 15368 tablet_replica.cc:330] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: stopping tablet replica
I20260620 02:39:08.489545 15368 raft_consensus.cc:2229] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:39:08.490316 15368 raft_consensus.cc:2258] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:08.492841 15368 ts_tablet_manager.cc:1905] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:39:08.499251 15369 tablet_replica.cc:330] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: stopping tablet replica
I20260620 02:39:08.499980 15369 raft_consensus.cc:2229] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:39:08.500622 15369 raft_consensus.cc:2258] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:08.501418 15367 ts_tablet_manager.cc:1905] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:39:08.503050 15369 ts_tablet_manager.cc:1905] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:39:08.506084 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.484446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.0.97.1:36939)
0620 02:39:08.484691 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.484700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.484725 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:39:08.484806 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.484814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.484948 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.485073 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:08.487975 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.489164 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.491708 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.492309 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.492475 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.496874 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:08.496898 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:08.496924 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:08.496987 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:08.501620 (+ 4633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:08.502751 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:08.502761 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:08.502765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:08.502959 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:08.503466 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:08.503475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:08.503478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:08.504087 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:08.504414 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.505094 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.505386 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:39:08.507624 15060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:08.511925 15369 ts_tablet_manager.cc:1918] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:08.511956 15368 ts_tablet_manager.cc:1918] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:08.512131 15369 log.cc:1196] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/ffe96a9fdd4f4f7fac36e0a7b015c7c9
I20260620 02:39:08.512177 15368 log.cc:1196] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/ffe96a9fdd4f4f7fac36e0a7b015c7c9
I20260620 02:39:08.512712 15369 ts_tablet_manager.cc:1939] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 2284be557f834092b535956fcf7b5eea: Deleting consensus metadata
I20260620 02:39:08.512811 15368 ts_tablet_manager.cc:1939] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 42db81b4b7aa474ea0042bd0f2d87fd4: Deleting consensus metadata
I20260620 02:39:08.514814 14864 catalog_manager.cc:4841] TS 42db81b4b7aa474ea0042bd0f2d87fd4 (127.0.97.2:42081): tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9 (table test-table [id=fa337788e4b1485383717cb8247a04c7]) successfully deleted
I20260620 02:39:08.515179 14865 catalog_manager.cc:4841] TS 2284be557f834092b535956fcf7b5eea (127.0.97.3:44103): tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9 (table test-table [id=fa337788e4b1485383717cb8247a04c7]) successfully deleted
I20260620 02:39:08.604543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 14917
I20260620 02:39:08.624388 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 15069
W20260620 02:39:08.624572 14866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36939 (error 108)
I20260620 02:39:08.626861 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.626304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36939 (local address 127.0.0.1:54096)
0620 02:39:08.626489 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.626633 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 02:39:08.627216 14866 catalog_manager.cc:4584] TS 7175767fdecd49e3a35b46f248578b8f (127.0.97.1:36939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffe96a9fdd4f4f7fac36e0a7b015c7c9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36939: connect: Connection refused (error 111)
I20260620 02:39:08.643654 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.626012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.0.97.2:42081)
0620 02:39:08.626210 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.626216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.626240 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:39:08.626370 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.626377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.626503 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.626637 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:08.628599 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.629561 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.631706 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.632367 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.632525 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.636181 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:08.636198 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:08.636214 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:08.636261 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:08.639724 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:08.640909 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:08.640917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:08.640920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:08.641078 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:08.641705 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:08.641710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:08.641713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:08.642215 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:08.642523 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.642982 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.643189 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:39:08.644578 15192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:08.680912 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.680459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36939 (local address 127.0.0.1:54104)
0620 02:39:08.680673 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.680770 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 02:39:08.682194 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.681785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36939 (local address 127.0.0.1:54114)
0620 02:39:08.681972 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.682056 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 02:39:08.743345 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.742697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36939 (local address 127.0.0.1:54116)
0620 02:39:08.742941 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.743056 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 02:39:08.744614 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.744259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36939 (local address 127.0.0.1:54132)
0620 02:39:08.744409 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.744487 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 02:39:08.766093 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 15069
I20260620 02:39:08.786005 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 15200
I20260620 02:39:08.803431 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.787193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.0.97.3:44103)
0620 02:39:08.787374 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:08.787378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:08.787397 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:39:08.787453 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:08.787457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:08.787580 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:08.787680 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:08.789419 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.790231 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.791702 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:08.792334 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:08.792455 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:08.795395 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:08.795422 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:08.795440 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:08.795487 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:08.798752 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:08.799648 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:08.799657 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:08.799659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:08.800101 (+ 442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:08.801071 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:08.801077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:08.801079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:08.801522 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:08.802149 (+ 627us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:08.802785 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 02:39:08.803017 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:39:08.804244 15323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:08.825697 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.825074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36939 (local address 127.0.0.1:54146)
0620 02:39:08.825340 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.825460 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 02:39:08.826987 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:08.826496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36939 (local address 127.0.0.1:54158)
0620 02:39:08.826740 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:08.826843 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 02:39:08.897477 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 15200
I20260620 02:39:08.916496 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 14847
I20260620 02:39:08.917919 14910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:09.039126 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 14847
I20260620 02:39:09.063222 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37319
--webserver_interface=127.0.97.62
--webserver_port=45153
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:09.421759 15375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:09.422159 15375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:09.422330 15375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:09.430277 15375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:39:09.430487 15375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:09.430610 15375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:09.430720 15375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:39:09.430815 15375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:39:09.441658 15375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:09.443174 15375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:09.444592 15375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:09.457656 15383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:09.457751 15381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:09.457711 15380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:09.458662 15375 server_base.cc:1034] running on GCE node
I20260620 02:39:09.459625 15375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:09.461392 15375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:09.462671 15375 hybrid_clock.cc:648] HybridClock initialized: now 1781923149462614 us; error 55 us; skew 500 ppm
I20260620 02:39:09.463074 15375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:09 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:09.466717 15375 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:39:09.468504 15375 webserver.cc:460] Webserver started at http://127.0.97.62:45153/ using document root <none> and password file <none>
I20260620 02:39:09.469223 15375 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:09.469329 15375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:09.475114 15375 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:39:09.480369 15390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:09.482101 15375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 02:39:09.482287 15375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3"
format_stamp: "Formatted at 2026-06-20 02:39:02 on dist-test-slave-rpg6"
server_key: "3bf9fce644a60c2a9b094ee90c375fd3"
server_key_iv: "fdaba8ca6946450cc5e27e42abfeddbd"
server_key_version: "encryptionkey@0"
I20260620 02:39:09.483345 15375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:09.499336 15375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:09.504473 15375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:09.504854 15375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:09.526556 15375 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37319
I20260620 02:39:09.526593 15441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37319 every 8 connection(s)
I20260620 02:39:09.529127 15375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:39:09.531569 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 15375
I20260620 02:39:09.532732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36939
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37319
--webserver_port=35623
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:39:09.538622 15442 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:39:09.542316 15442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Bootstrap starting.
I20260620 02:39:09.565186 15442 log.cc:824] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:09.582444 15442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:39:09.583076 15442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3: Bootstrap complete.
I20260620 02:39:09.594187 15442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:09.595237 15442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ebf31f3bb34d23a0afec81d5eb1fc3, State: Initialized, Role: FOLLOWER
I20260620 02:39:09.595968 15442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:09.596246 15442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:39:09.596391 15442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:39:09.596616 15442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:39:09.600939 15442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:09.601536 15442 leader_election.cc:302] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52ebf31f3bb34d23a0afec81d5eb1fc3; no voters:
I20260620 02:39:09.602274 15442 leader_election.cc:288] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 02:39:09.602424 15446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 02:39:09.603755 15446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [term 3 LEADER]: Becoming Leader. State: Replica: 52ebf31f3bb34d23a0afec81d5eb1fc3, State: Running, Role: LEADER
I20260620 02:39:09.604512 15446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } }
I20260620 02:39:09.604784 15442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:39:09.607342 15448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ebf31f3bb34d23a0afec81d5eb1fc3. Latest consensus state: current_term: 3 leader_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } } }
I20260620 02:39:09.607771 15448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:09.608294 15447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ebf31f3bb34d23a0afec81d5eb1fc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37319 } } }
I20260620 02:39:09.608562 15447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ebf31f3bb34d23a0afec81d5eb1fc3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:09.611505 15454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:39:09.623126 15454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:39:09.625546 15454 catalog_manager.cc:1259] Loaded cluster ID: 8dd9974dad7b466989fa5033525b38ee
I20260620 02:39:09.625727 15454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 02:39:09.628739 15454 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f67c8fec9ad google::LogMessage::Fail() at ??:0
@ 0x7f67c8fef88a google::LogMessage::SendToLog() at ??:0
@ 0x7f67c8fec4de google::LogMessage::Flush() at ??:0
I20260620 02:39:09.638701 15463 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f67c8feda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f67dea23512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f67dea20976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f67deaa8610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f67deaa8439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f67cbc38345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f67cbc3b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f67cbc3b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f67cbc16eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f67ceef06db start_thread at ??:0
@ 0x7f67c7e8661f clone at ??:0
W20260620 02:39:09.967628 15444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:09.968055 15444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:09.968168 15444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:09.975931 15444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:09.976078 15444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:09.976207 15444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:39:09.986758 15444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35623
--tserver_master_addrs=127.0.97.62:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:09.988233 15444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:09.989566 15444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:10.005009 15486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:10.004978 15484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:10.005321 15483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:10.006022 15444 server_base.cc:1034] running on GCE node
I20260620 02:39:10.006732 15444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:10.007819 15444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:10.009155 15444 hybrid_clock.cc:648] HybridClock initialized: now 1781923150009071 us; error 98 us; skew 500 ppm
I20260620 02:39:10.009760 15444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:10 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:10.014235 15444 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:39:10.016647 15444 webserver.cc:460] Webserver started at http://127.0.97.1:35623/ using document root <none> and password file <none>
I20260620 02:39:10.017498 15444 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:10.017623 15444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:10.024042 15444 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 02:39:10.027820 15493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:10.029876 15444 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:39:10.030115 15444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "7175767fdecd49e3a35b46f248578b8f"
format_stamp: "Formatted at 2026-06-20 02:39:03 on dist-test-slave-rpg6"
server_key: "d6a9aa1b087ff9bff18bb84d91aebd45"
server_key_iv: "048e748175302dad63494cf3fb5bb21c"
server_key_version: "encryptionkey@0"
I20260620 02:39:10.031193 15444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:10.062526 15444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:10.067597 15444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:10.067965 15444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:10.069080 15444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:10.071251 15500 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 02:39:10.078709 15500 ts_tablet_manager.cc:1852] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 02:39:10.080049 15500 ts_tablet_manager.cc:1905] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:39:10.088307 15500 ts_tablet_manager.cc:1918] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:10.088613 15500 log.cc:1196] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/ffe96a9fdd4f4f7fac36e0a7b015c7c9
I20260620 02:39:10.089182 15500 ts_tablet_manager.cc:1939] T ffe96a9fdd4f4f7fac36e0a7b015c7c9 P 7175767fdecd49e3a35b46f248578b8f: Deleting consensus metadata
I20260620 02:39:10.089675 15444 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 02:39:10.089792 15444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.019s user 0.001s sys 0.000s
I20260620 02:39:10.089895 15444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:10.089983 15444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:10.128312 15444 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36939
I20260620 02:39:10.128669 15606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36939 every 8 connection(s)
I20260620 02:39:10.129916 15444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:39:10.131376 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 15444
I20260620 02:39:10.132262 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42081
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37319
--webserver_port=39477
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:39:10.136351 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:10.133912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:42251)
0620 02:39:10.134944 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:10.135207 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":367,"threads_started":1}
W20260620 02:39:10.137212 15607 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37319 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37319: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
I20260620 02:39:10.138285 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:10.137889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:35583)
0620 02:39:10.138078 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:10.138167 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:39:10.139410 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:10.139035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:36669)
0620 02:39:10.139198 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:10.139279 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 02:39:10.139896 15607 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:39:10.492107 15610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:10.492499 15610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:10.492677 15610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:10.500399 15610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:10.500597 15610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:10.500813 15610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:39:10.511385 15610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39477
--tserver_master_addrs=127.0.97.62:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:10.512810 15610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:10.514187 15610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:10.528059 15618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:10.528129 15616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:10.528170 15615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:10.528918 15610 server_base.cc:1034] running on GCE node
I20260620 02:39:10.529632 15610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:10.530666 15610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:10.531960 15610 hybrid_clock.cc:648] HybridClock initialized: now 1781923150531905 us; error 44 us; skew 500 ppm
I20260620 02:39:10.532317 15610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:10 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:10.536149 15610 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:39:10.537941 15610 webserver.cc:460] Webserver started at http://127.0.97.2:39477/ using document root <none> and password file <none>
I20260620 02:39:10.538787 15610 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:10.538949 15610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:10.544925 15610 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:39:10.548319 15625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:10.550082 15610 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260620 02:39:10.550318 15610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "42db81b4b7aa474ea0042bd0f2d87fd4"
format_stamp: "Formatted at 2026-06-20 02:39:04 on dist-test-slave-rpg6"
server_key: "ecfffbb8962cdcc53e3b78c854ade989"
server_key_iv: "6d79f7d347c8cc6833d2cf169c7ca4fa"
server_key_version: "encryptionkey@0"
I20260620 02:39:10.551448 15610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:10.571403 15610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:10.576547 15610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:10.576884 15610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:10.577956 15610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:10.579581 15610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:10.579708 15610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:10.579872 15610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:10.579984 15610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 02:39:10.617549 15610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42081
I20260620 02:39:10.617609 15737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42081 every 8 connection(s)
I20260620 02:39:10.619130 15610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:39:10.620527 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 15610
I20260620 02:39:10.621287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44103
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37319
--webserver_port=41097
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:39:10.625937 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:10.622964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:60893)
0620 02:39:10.624396 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:10.624646 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":499,"threads_started":1}
W20260620 02:39:10.626827 15738 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37319 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37319: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
I20260620 02:39:10.628080 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:10.627518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:37625)
0620 02:39:10.627758 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:10.627893 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:39:10.629207 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:10.628763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:49987)
0620 02:39:10.628964 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:10.629084 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 02:39:10.629534 15738 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:39:10.976285 15741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:10.976682 15741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:10.976862 15741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:10.984613 15741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:10.984797 15741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:10.984993 15741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:39:10.995360 15741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41097
--tserver_master_addrs=127.0.97.62:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:10.996881 15741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:10.998592 15741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:11.013199 15749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:11.013245 15746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:11.013378 15747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:11.014081 15741 server_base.cc:1034] running on GCE node
I20260620 02:39:11.014694 15741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:11.015728 15741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:11.017028 15741 hybrid_clock.cc:648] HybridClock initialized: now 1781923151016948 us; error 75 us; skew 500 ppm
I20260620 02:39:11.017418 15741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:11 dist-test-slave-rpg6 krb5kdc[14292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:11.020872 15741 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:39:11.022670 15741 webserver.cc:460] Webserver started at http://127.0.97.3:41097/ using document root <none> and password file <none>
I20260620 02:39:11.023379 15741 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:11.023485 15741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:11.029449 15741 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:39:11.032577 15756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:11.034278 15741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:39:11.034497 15741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "2284be557f834092b535956fcf7b5eea"
format_stamp: "Formatted at 2026-06-20 02:39:04 on dist-test-slave-rpg6"
server_key: "3908758f42c130a51356a58324ebd0e2"
server_key_iv: "e49c8ff440ddae46e1fd56a92941278d"
server_key_version: "encryptionkey@0"
I20260620 02:39:11.035742 15741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:11.052378 15741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:11.057349 15741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:11.057613 15741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:11.058812 15741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:11.060578 15741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:11.060717 15741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:11.060843 15741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:11.060927 15741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:11.108270 15741 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44103
I20260620 02:39:11.108371 15868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44103 every 8 connection(s)
I20260620 02:39:11.109930 15741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:39:11.116428 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:11.114193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:39015)
0620 02:39:11.115065 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:11.115291 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":361,"threads_started":1}
W20260620 02:39:11.117303 15869 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37319 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37319: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
I20260620 02:39:11.118543 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:11.117950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:57587)
0620 02:39:11.118300 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:11.118400 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260620 02:39:11.120260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 15741
I20260620 02:39:11.120338 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:11.119639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:38647)
0620 02:39:11.119925 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:11.120155 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260620 02:39:11.120615 15869 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:39:11.141901 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:11.140889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:33375)
0620 02:39:11.141484 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:11.141598 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":250,"threads_started":1}
I20260620 02:39:11.631279 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:11.630208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:44013)
0620 02:39:11.630775 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:11.630882 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":292,"threads_started":1}
I20260620 02:39:12.122325 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:12.121291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:52329)
0620 02:39:12.121969 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:12.122078 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":379,"threads_started":1}
I20260620 02:39:12.144124 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:12.143154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:40747)
0620 02:39:12.143675 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:12.143774 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":271,"threads_started":1}
I20260620 02:39:12.633373 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:12.632517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:46509)
0620 02:39:12.633075 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:12.633185 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":242,"threads_started":1}
I20260620 02:39:13.124390 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:13.123466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:52323)
0620 02:39:13.124076 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:13.124188 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":265,"threads_started":1}
I20260620 02:39:13.145886 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:13.145107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:47345)
0620 02:39:13.145591 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:13.145700 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":245,"threads_started":1}
I20260620 02:39:13.635340 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:13.634495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:39737)
0620 02:39:13.635062 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:13.635153 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":338,"threads_started":1}
I20260620 02:39:14.126845 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:14.125623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:44965)
0620 02:39:14.126456 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:14.126591 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":301,"threads_started":1}
I20260620 02:39:14.148135 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:14.147059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:53333)
0620 02:39:14.147602 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:14.147695 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":283,"threads_started":1}
I20260620 02:39:14.637523 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:14.636695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:34863)
0620 02:39:14.637246 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:14.637330 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
I20260620 02:39:15.128947 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:15.128106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:60381)
0620 02:39:15.128631 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:15.128742 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":248,"threads_started":1}
I20260620 02:39:15.150058 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:15.149202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:58865)
0620 02:39:15.149790 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:15.149879 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":341,"threads_started":1}
I20260620 02:39:15.639724 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:15.638757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:58025)
0620 02:39:15.639405 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:15.639510 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":373,"threads_started":1}
I20260620 02:39:16.131048 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:16.130131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:40457)
0620 02:39:16.130789 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:16.130878 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":359,"threads_started":1}
I20260620 02:39:16.152169 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:16.151182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:41653)
0620 02:39:16.151714 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:16.151805 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":265,"threads_started":1}
I20260620 02:39:16.641980 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:16.640996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:49067)
0620 02:39:16.641614 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:16.641749 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":311,"threads_started":1}
I20260620 02:39:17.132973 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:17.132111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:54685)
0620 02:39:17.132697 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:17.132799 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":300,"threads_started":1}
I20260620 02:39:17.154407 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:17.153416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:59797)
0620 02:39:17.154048 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:17.154158 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":308,"threads_started":1}
I20260620 02:39:17.644054 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:17.643046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:60861)
0620 02:39:17.643607 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:17.643696 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":270,"threads_started":1}
I20260620 02:39:18.135694 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:18.134268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:40285)
0620 02:39:18.135296 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:18.135424 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":312,"threads_started":1}
I20260620 02:39:18.156519 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:18.155579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:34229)
0620 02:39:18.156249 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:18.156339 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":332,"threads_started":1}
I20260620 02:39:18.645869 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:18.645044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:37397)
0620 02:39:18.645603 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:18.645693 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":299,"threads_started":1}
I20260620 02:39:19.137656 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:19.136834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:40383)
0620 02:39:19.137350 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:19.137456 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":261,"threads_started":1}
I20260620 02:39:19.158527 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:19.157640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:48673)
0620 02:39:19.158237 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:19.158348 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":297,"threads_started":1}
I20260620 02:39:19.648087 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:19.647079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:42177)
0620 02:39:19.647712 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:19.647805 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":267,"threads_started":1}
I20260620 02:39:20.139767 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:20.138853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:38821)
0620 02:39:20.139393 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:20.139483 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":271,"threads_started":1}
I20260620 02:39:20.160475 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:20.159568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:44039)
0620 02:39:20.160205 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:20.160298 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":335,"threads_started":1}
I20260620 02:39:20.650049 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:20.649279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:45261)
0620 02:39:20.649794 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:20.649883 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":294,"threads_started":1}
I20260620 02:39:21.141702 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:21.140859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:58141)
0620 02:39:21.141442 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:21.141534 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":294,"threads_started":1}
I20260620 02:39:21.162340 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:21.161462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:39211)
0620 02:39:21.162063 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:21.162151 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":364,"threads_started":1}
I20260620 02:39:21.652130 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:21.651259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:37903)
0620 02:39:21.651736 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:21.651906 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":246,"threads_started":1}
I20260620 02:39:22.143643 15905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:22.142828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:42627)
0620 02:39:22.143314 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:22.143416 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":254,"threads_started":1}
I20260620 02:39:22.164191 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:22.163377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:39715)
0620 02:39:22.163923 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:22.164014 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":259,"threads_started":1}
I20260620 02:39:22.654158 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:22.653341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:60181)
0620 02:39:22.653865 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:22.653957 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":288,"threads_started":1}
I20260620 02:39:23.145426 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:23.144657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:44463)
0620 02:39:23.145178 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:23.145263 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":258,"threads_started":1}
I20260620 02:39:23.166596 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:23.165660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:33541)
0620 02:39:23.166264 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:23.166362 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":346,"threads_started":1}
I20260620 02:39:23.656998 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:23.655448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:55803)
0620 02:39:23.656571 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:23.656721 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":514,"threads_started":1}
I20260620 02:39:24.147312 15911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:24.146452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:41055)
0620 02:39:24.146995 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:24.147091 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":266,"threads_started":1}
I20260620 02:39:24.168551 15912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:24.167681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:36827)
0620 02:39:24.168268 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:24.168359 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":298,"threads_started":1}
I20260620 02:39:24.659125 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:24.658211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:44075)
0620 02:39:24.658812 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:24.658914 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":329,"threads_started":1}
I20260620 02:39:25.149341 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:25.148517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:46249)
0620 02:39:25.149034 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:25.149142 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":246,"threads_started":1}
I20260620 02:39:25.170709 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:25.169930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:45949)
0620 02:39:25.170420 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:25.170519 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":256,"threads_started":1}
I20260620 02:39:25.661413 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:25.660534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.2:38233)
0620 02:39:25.661124 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:25.661224 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":280,"threads_started":1}
I20260620 02:39:26.133813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 15444
I20260620 02:39:26.152774 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:26.151670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.3:37243)
0620 02:39:26.152468 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:26.152571 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":485,"threads_started":1}
I20260620 02:39:26.165230 15917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:26.135164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.0.97.1:36939)
0620 02:39:26.136065 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:26.136087 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:26.136610 (+ 523us) server_negotiation.cc:407] Connection header received
0620 02:39:26.139810 (+ 3200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:26.139885 (+ 75us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:26.140450 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:26.140867 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:26.142549 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:26.143944 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:26.145334 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:26.146143 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:26.146389 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:26.154566 (+ 8177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:26.154584 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:26.154611 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:26.154963 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:26.159260 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:26.160564 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:26.160579 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:26.160592 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:26.160845 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:26.161836 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:26.161842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:26.161845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:26.162701 (+ 856us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:26.162984 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:26.164217 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0620 02:39:26.164848 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":339,"threads_started":1}
I20260620 02:39:26.168553 15602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:26.268221 15920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:26.265441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37319 (local address 127.0.97.1:58909)
0620 02:39:26.267753 (+ 2312us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:26.267944 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2162,"thread_start_us":1917,"threads_started":1}
I20260620 02:39:26.269346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 15444
I20260620 02:39:26.290169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 15610
I20260620 02:39:26.321714 15921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:26.291908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.0.97.2:42081)
0620 02:39:26.292750 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:26.292773 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:26.292835 (+ 62us) server_negotiation.cc:407] Connection header received
0620 02:39:26.295419 (+ 2584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:26.295444 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:26.296269 (+ 825us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:26.296877 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:26.298937 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:26.301088 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:26.302934 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:26.304983 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:26.305588 (+ 605us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:26.309581 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:26.309591 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:26.309613 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:26.309668 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:26.314304 (+ 4636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:26.316370 (+ 2066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:26.316386 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:26.316421 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:26.316781 (+ 360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:26.318456 (+ 1675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:26.318462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:26.318466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:26.319420 (+ 954us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:26.319738 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:26.320809 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0620 02:39:26.321383 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":367,"threads_started":1}
I20260620 02:39:26.324074 15733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:26.463614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 15610
I20260620 02:39:26.500305 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 15741
I20260620 02:39:26.536422 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:26.501791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.0.97.3:44103)
0620 02:39:26.502532 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:26.502547 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:26.503013 (+ 466us) server_negotiation.cc:407] Connection header received
0620 02:39:26.505249 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:26.505263 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:26.505825 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:26.506247 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:26.510989 (+ 4742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:26.513514 (+ 2525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:26.516099 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:26.517572 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:26.517938 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:26.525357 (+ 7419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:26.525371 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:26.525392 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:26.525454 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:26.530265 (+ 4811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:26.532467 (+ 2202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:26.532479 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:26.532487 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:26.532794 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:26.533761 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:26.533766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:26.533768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:26.534520 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:26.534812 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:26.535522 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 02:39:26.536077 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":393,"threads_started":1}
I20260620 02:39:26.538158 15864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:26.678990 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 15741
2026-06-20T02:39:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25599 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:39:26 dist-test-slave-rpg6 krb5kdc[15927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:39:26 dist-test-slave-rpg6 krb5kdc[15927](info): set up 2 sockets
Jun 20 02:39:26 dist-test-slave-rpg6 krb5kdc[15927](info): commencing operation
krb5kdc: starting...
W20260620 02:39:27.928961 388 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:39:28 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:39:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:39:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:39:28.228760 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41621
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:28.605909 15940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:28.606271 15940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:28.606431 15940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:28.614318 15940 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:39:28.614465 15940 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:39:28.614506 15940 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:39:28.614605 15940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:39:28.614712 15940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:28.614782 15940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:28.614885 15940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:39:28.614948 15940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:39:28.625725 15940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:28.627286 15940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:28.628811 15940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:28.642731 15946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:28.642737 15948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:28.642784 15945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:28.643169 15940 server_base.cc:1034] running on GCE node
I20260620 02:39:28.644555 15940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:28.646278 15940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:28.647572 15940 hybrid_clock.cc:648] HybridClock initialized: now 1781923168647501 us; error 52 us; skew 500 ppm
I20260620 02:39:28.648154 15940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:28 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:28.652419 15940 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:39:28.654325 15940 webserver.cc:460] Webserver started at http://127.0.97.62:39503/ using document root <none> and password file <none>
I20260620 02:39:28.655133 15940 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:28.655279 15940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:28.655629 15940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:39:28.659037 15940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "57cb051a75554f298208f7cea386dc10"
format_stamp: "Formatted at 2026-06-20 02:39:28 on dist-test-slave-rpg6"
server_key: "ff5f8c6ae0e6298008dc9a4e552e2640"
server_key_iv: "74001b4eebfab1588b0a4c93383cc08c"
server_key_version: "encryptionkey@0"
I20260620 02:39:28.660004 15940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "57cb051a75554f298208f7cea386dc10"
format_stamp: "Formatted at 2026-06-20 02:39:28 on dist-test-slave-rpg6"
server_key: "ff5f8c6ae0e6298008dc9a4e552e2640"
server_key_iv: "74001b4eebfab1588b0a4c93383cc08c"
server_key_version: "encryptionkey@0"
I20260620 02:39:28.666399 15940 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:39:28.672683 15955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:28.674729 15940 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260620 02:39:28.675017 15940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "57cb051a75554f298208f7cea386dc10"
format_stamp: "Formatted at 2026-06-20 02:39:28 on dist-test-slave-rpg6"
server_key: "ff5f8c6ae0e6298008dc9a4e552e2640"
server_key_iv: "74001b4eebfab1588b0a4c93383cc08c"
server_key_version: "encryptionkey@0"
I20260620 02:39:28.675335 15940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:28.692924 15940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:28.697911 15940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:28.698246 15940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:28.720995 15940 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41621
I20260620 02:39:28.721097 16006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41621 every 8 connection(s)
I20260620 02:39:28.722532 15940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:39:28.726948 16007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:28.728345 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 15940
I20260620 02:39:28.728613 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:39:28.729480 388 external_mini_cluster.cc:1428] Setting key d575a640cacc03aa22f6b0647f040c6a
I20260620 02:39:28.744386 16007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Bootstrap starting.
Jun 20 02:39:28 dist-test-slave-rpg6 krb5kdc[15927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:28.749013 16007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:28.750644 16007 log.cc:824] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:28.755406 16007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: No bootstrap required, opened a new log
I20260620 02:39:28.760144 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:28.732432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.0.97.62:41621)
0620 02:39:28.733469 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:28.733495 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:28.733543 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:39:28.735293 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:28.735324 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:28.735820 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:28.736332 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:28.738611 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:28.740215 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:28.742357 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:28.743611 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:28.743784 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:28.749336 (+ 5552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:28.749350 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:28.749373 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:28.749444 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:28.754375 (+ 4931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:28.755533 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:28.755542 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:28.755552 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:28.755790 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:28.756368 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:28.756373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:28.756377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:28.757141 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:28.757452 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:28.757976 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 02:39:28.758511 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":547,"threads_started":1}
I20260620 02:39:28.766503 16007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:39:28.766971 16007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:28.767053 16007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cb051a75554f298208f7cea386dc10, State: Initialized, Role: FOLLOWER
I20260620 02:39:28.767719 16007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:39:28.768021 16007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:39:28.768129 16007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:39:28.768260 16007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:28.771015 16007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:39:28.771536 16007 leader_election.cc:302] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57cb051a75554f298208f7cea386dc10; no voters:
I20260620 02:39:28.772238 16007 leader_election.cc:288] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:39:28.772413 16012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:39:28.773245 16012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 1 LEADER]: Becoming Leader. State: Replica: 57cb051a75554f298208f7cea386dc10, State: Running, Role: LEADER
I20260620 02:39:28.773860 16012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:39:28.774850 16007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:39:28.777182 16014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57cb051a75554f298208f7cea386dc10. Latest consensus state: current_term: 1 leader_uuid: "57cb051a75554f298208f7cea386dc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } } }
I20260620 02:39:28.777655 16014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:28.778527 16013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57cb051a75554f298208f7cea386dc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } } }
I20260620 02:39:28.779029 16013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:28.780037 16019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:39:28.787350 16019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:39:28.799573 16019 catalog_manager.cc:1347] Generated new cluster ID: c83ec8c871464ffda9da705cbdba25c1
I20260620 02:39:28.799739 16019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:39:28.815230 16019 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:39:28.816784 16019 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:39:28.828156 16019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Generated new TSK 0
I20260620 02:39:28.829427 16019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:39:28.910609 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41621
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:29.296774 16033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:29.297115 16033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:29.297284 16033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:29.305364 16033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:29.305517 16033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:29.305713 16033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:39:29.316953 16033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:29.318498 16033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:29.319965 16033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:29.335681 16038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:29.335623 16041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:29.335700 16039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:29.336308 16033 server_base.cc:1034] running on GCE node
I20260620 02:39:29.337049 16033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:29.338231 16033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:29.339509 16033 hybrid_clock.cc:648] HybridClock initialized: now 1781923169339424 us; error 76 us; skew 500 ppm
I20260620 02:39:29.339951 16033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:29 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:29.343617 16033 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:39:29.345368 16033 webserver.cc:460] Webserver started at http://127.0.97.1:37463/ using document root <none> and password file <none>
I20260620 02:39:29.346127 16033 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:29.346240 16033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:29.346527 16033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:39:29.349860 16033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "141cb494da5b44cf89d2ba0d4402f7e1"
format_stamp: "Formatted at 2026-06-20 02:39:29 on dist-test-slave-rpg6"
server_key: "0d40857a506fe6b5d01f5e7d3891cb45"
server_key_iv: "62cc683e3260cff4894f3df23f3b1317"
server_key_version: "encryptionkey@0"
I20260620 02:39:29.350668 16033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "141cb494da5b44cf89d2ba0d4402f7e1"
format_stamp: "Formatted at 2026-06-20 02:39:29 on dist-test-slave-rpg6"
server_key: "0d40857a506fe6b5d01f5e7d3891cb45"
server_key_iv: "62cc683e3260cff4894f3df23f3b1317"
server_key_version: "encryptionkey@0"
I20260620 02:39:29.356586 16033 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:39:29.361119 16048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:29.362977 16033 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:39:29.363166 16033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "141cb494da5b44cf89d2ba0d4402f7e1"
format_stamp: "Formatted at 2026-06-20 02:39:29 on dist-test-slave-rpg6"
server_key: "0d40857a506fe6b5d01f5e7d3891cb45"
server_key_iv: "62cc683e3260cff4894f3df23f3b1317"
server_key_version: "encryptionkey@0"
I20260620 02:39:29.363391 16033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:29.375561 16033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:29.380934 16033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:29.381384 16033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:29.382577 16033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:29.384303 16033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:29.384424 16033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:29.384596 16033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:29.384671 16033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:29.423307 16033 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34435
I20260620 02:39:29.423327 16160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34435 every 8 connection(s)
I20260620 02:39:29.424894 16033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:39:29.429075 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 16033
I20260620 02:39:29.429589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:39:29.430541 388 external_mini_cluster.cc:1428] Setting key 276aaf507a45cc9ffa35745712bbe16f
Jun 20 02:39:29 dist-test-slave-rpg6 krb5kdc[15927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:29.452211 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:29.428044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43221 (local address 127.0.97.62:41621)
0620 02:39:29.428990 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:29.428999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:29.430137 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 02:39:29.432785 (+ 2648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:29.432790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:29.432873 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:29.433020 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:29.435037 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:29.436910 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:29.439011 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:29.439779 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:29.439998 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:29.444737 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:29.444743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:29.444747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:29.444788 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:29.447784 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:29.449150 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:29.449155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:29.449157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:29.449303 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:29.449899 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:29.449903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:29.449905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:29.450286 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:29.450407 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:29.451501 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0620 02:39:29.451732 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":507,"threads_started":1}
I20260620 02:39:29.453539 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:29.428795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:43221)
0620 02:39:29.429880 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:29.429952 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:29.432428 (+ 2476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:29.433354 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:29.433364 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:29.434034 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:29.434700 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:29.434721 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:29.437227 (+ 2506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:29.437235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:29.438765 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:29.438784 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:29.440094 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:29.440102 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:29.440364 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:29.441215 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:29.441241 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:29.444380 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:29.448075 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:29.448089 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:29.448113 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:29.448892 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:29.449475 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:29.449483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:29.449490 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:29.449771 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:29.450470 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:29.450483 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:29.451003 (+ 520us) client_negotiation.cc:769] Sending connection context
0620 02:39:29.451343 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 02:39:29.452016 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":475,"threads_started":1}
I20260620 02:39:29.455469 16161 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41621
I20260620 02:39:29.455907 16161 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:29.456780 16161 heartbeater.cc:511] Master 127.0.97.62:41621 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:39:29.460125 15972 ts_manager.cc:194] Registered new tserver with Master: 141cb494da5b44cf89d2ba0d4402f7e1 (127.0.97.1:34435)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:39:29.462244 15972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:39:29.496428 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41621
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:29.862286 16167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:29.862655 16167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:29.862828 16167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:29.871300 16167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:29.871501 16167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:29.871690 16167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:39:29.883195 16167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:29.884753 16167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:29.886174 16167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:29.901494 16175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:29.901657 16173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:29.901733 16172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:29.903389 16167 server_base.cc:1034] running on GCE node
I20260620 02:39:29.904371 16167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:29.905654 16167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:29.906903 16167 hybrid_clock.cc:648] HybridClock initialized: now 1781923169906846 us; error 56 us; skew 500 ppm
I20260620 02:39:29.907332 16167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:29 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:29.911187 16167 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:39:29.913024 16167 webserver.cc:460] Webserver started at http://127.0.97.2:35377/ using document root <none> and password file <none>
I20260620 02:39:29.913887 16167 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:29.914047 16167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:29.914433 16167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:39:29.917842 16167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "884c4079aa784968bb1df80a930599c3"
format_stamp: "Formatted at 2026-06-20 02:39:29 on dist-test-slave-rpg6"
server_key: "631a7ddc9558ceff0527ac9b6e9a88ed"
server_key_iv: "647b0f46050141b0bf2de229ced4284d"
server_key_version: "encryptionkey@0"
I20260620 02:39:29.918769 16167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "884c4079aa784968bb1df80a930599c3"
format_stamp: "Formatted at 2026-06-20 02:39:29 on dist-test-slave-rpg6"
server_key: "631a7ddc9558ceff0527ac9b6e9a88ed"
server_key_iv: "647b0f46050141b0bf2de229ced4284d"
server_key_version: "encryptionkey@0"
I20260620 02:39:29.925139 16167 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:39:29.930001 16182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:29.931989 16167 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260620 02:39:29.932276 16167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "884c4079aa784968bb1df80a930599c3"
format_stamp: "Formatted at 2026-06-20 02:39:29 on dist-test-slave-rpg6"
server_key: "631a7ddc9558ceff0527ac9b6e9a88ed"
server_key_iv: "647b0f46050141b0bf2de229ced4284d"
server_key_version: "encryptionkey@0"
I20260620 02:39:29.932524 16167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:29.952188 16167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:29.959004 16167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:29.959399 16167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:29.960685 16167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:29.962384 16167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:29.962467 16167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:29.962612 16167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:29.962664 16167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:30.002733 16167 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36877
I20260620 02:39:30.002867 16294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36877 every 8 connection(s)
I20260620 02:39:30.004341 16167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:39:30.013468 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 16167
I20260620 02:39:30.013854 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:39:30.014694 388 external_mini_cluster.cc:1428] Setting key 493057f6bf72e4d52f0d86b144b0a2c7
Jun 20 02:39:30 dist-test-slave-rpg6 krb5kdc[15927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:30.031968 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.007664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59411 (local address 127.0.97.62:41621)
0620 02:39:30.008381 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:30.008386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:30.009532 (+ 1146us) server_negotiation.cc:407] Connection header received
0620 02:39:30.011595 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:30.011601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:30.011710 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:30.011845 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:30.015637 (+ 3792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.016621 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.018626 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.019090 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.019251 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.023625 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:30.023631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:30.023634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:30.023672 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:30.026880 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:30.029005 (+ 2125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:30.029011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:30.029013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:30.029187 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:30.029874 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:30.029878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:30.029881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:30.030205 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:30.030360 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:30.031358 (+ 998us) server_negotiation.cc:299] Negotiation successful
0620 02:39:30.031574 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":495,"threads_started":1}
I20260620 02:39:30.033512 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.008343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:59411)
0620 02:39:30.009298 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:30.009351 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:30.011236 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:30.012545 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:30.012561 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:30.014206 (+ 1645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:30.015337 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.015378 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.016852 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.016857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.018417 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.018433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.019373 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.019380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.019651 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.020504 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:30.020532 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:30.023276 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:30.027324 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:30.027344 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:30.027376 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:30.028750 (+ 1374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:30.029336 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:30.029344 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:30.029351 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:30.029692 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:30.030358 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:30.030370 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:30.030897 (+ 527us) client_negotiation.cc:769] Sending connection context
0620 02:39:30.031237 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 02:39:30.031917 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":334,"threads_started":1}
I20260620 02:39:30.035320 16295 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41621
I20260620 02:39:30.035665 16295 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:30.036636 16295 heartbeater.cc:511] Master 127.0.97.62:41621 requested a full tablet report, sending...
I20260620 02:39:30.038341 15972 ts_manager.cc:194] Registered new tserver with Master: 884c4079aa784968bb1df80a930599c3 (127.0.97.2:36877)
I20260620 02:39:30.039232 15972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59411
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:39:30.081650 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41621
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:30.448566 16301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:30.449075 16301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:30.449398 16301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:30.458752 16301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:30.458983 16301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:30.459192 16301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:39:30.466259 16161 heartbeater.cc:503] Master 127.0.97.62:41621 was elected leader, sending a full tablet report...
I20260620 02:39:30.470348 16301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:30.471782 16301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:30.473258 16301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:30.488117 16307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:30.488186 16309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:30.488152 16306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:30.489043 16301 server_base.cc:1034] running on GCE node
I20260620 02:39:30.489830 16301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:30.490949 16301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:30.492233 16301 hybrid_clock.cc:648] HybridClock initialized: now 1781923170492166 us; error 57 us; skew 500 ppm
I20260620 02:39:30.492655 16301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:30 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:30.496367 16301 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:39:30.498317 16301 webserver.cc:460] Webserver started at http://127.0.97.3:36341/ using document root <none> and password file <none>
I20260620 02:39:30.499089 16301 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:30.499246 16301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:30.499591 16301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:39:30.503037 16301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "a66cc8ab6b504b3da58ad372da5dccfd"
format_stamp: "Formatted at 2026-06-20 02:39:30 on dist-test-slave-rpg6"
server_key: "24756062d51eab65ecb5c8d5c2085709"
server_key_iv: "a779e60235270f2de648e795d4f75b42"
server_key_version: "encryptionkey@0"
I20260620 02:39:30.504038 16301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a66cc8ab6b504b3da58ad372da5dccfd"
format_stamp: "Formatted at 2026-06-20 02:39:30 on dist-test-slave-rpg6"
server_key: "24756062d51eab65ecb5c8d5c2085709"
server_key_iv: "a779e60235270f2de648e795d4f75b42"
server_key_version: "encryptionkey@0"
I20260620 02:39:30.510968 16301 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:39:30.516698 16316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:30.518680 16301 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:39:30.518934 16301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "a66cc8ab6b504b3da58ad372da5dccfd"
format_stamp: "Formatted at 2026-06-20 02:39:30 on dist-test-slave-rpg6"
server_key: "24756062d51eab65ecb5c8d5c2085709"
server_key_iv: "a779e60235270f2de648e795d4f75b42"
server_key_version: "encryptionkey@0"
I20260620 02:39:30.519186 16301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:30.553156 16301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:30.558283 16301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:30.558615 16301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:30.559698 16301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:30.561324 16301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:30.561439 16301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:30.561606 16301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:30.561682 16301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:30.599277 16301 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44267
I20260620 02:39:30.599344 16428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44267 every 8 connection(s)
I20260620 02:39:30.600829 16301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:39:30.610100 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 16301
I20260620 02:39:30.610344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:39:30.610836 388 external_mini_cluster.cc:1428] Setting key 0e5f4a48ff34814fc69fe2ffe8227d23
Jun 20 02:39:30 dist-test-slave-rpg6 krb5kdc[15927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:30.630788 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.603821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52779 (local address 127.0.97.62:41621)
0620 02:39:30.604503 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:30.604508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:30.605620 (+ 1112us) server_negotiation.cc:407] Connection header received
0620 02:39:30.607773 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:30.607778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:30.607929 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:30.608059 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:30.609767 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.610687 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.612741 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.613491 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.613634 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.621120 (+ 7486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:30.621128 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:30.621132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:30.621203 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:30.625226 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:30.626521 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:30.626527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:30.626531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:30.626740 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:30.627937 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:30.627945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:30.627948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:30.628653 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:30.628882 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:30.630113 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0620 02:39:30.630433 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":286,"threads_started":1}
I20260620 02:39:30.631556 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.604477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:52779)
0620 02:39:30.605311 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:30.605353 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:30.607433 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:30.608339 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:30.608348 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:30.608949 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:30.609565 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.609583 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.610865 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.610869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.612582 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.612597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.613745 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.613754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.614022 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.616374 (+ 2352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:30.616432 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:30.620558 (+ 4126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:30.625462 (+ 4904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:30.625481 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:30.625502 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:30.626283 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:30.626991 (+ 708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:30.626998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:30.627002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:30.627596 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:30.628886 (+ 1290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:30.628899 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:30.629428 (+ 529us) client_negotiation.cc:769] Sending connection context
0620 02:39:30.629714 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 02:39:30.630220 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":341,"threads_started":1}
I20260620 02:39:30.633522 16429 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41621
I20260620 02:39:30.633968 16429 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:30.634805 16429 heartbeater.cc:511] Master 127.0.97.62:41621 requested a full tablet report, sending...
I20260620 02:39:30.636991 15972 ts_manager.cc:194] Registered new tserver with Master: a66cc8ab6b504b3da58ad372da5dccfd (127.0.97.3:44267)
I20260620 02:39:30.638019 15972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52779
I20260620 02:39:30.641669 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:39:30.663486 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.647244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.0.97.62:41621)
0620 02:39:30.647558 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:30.647566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:30.647937 (+ 371us) server_negotiation.cc:407] Connection header received
0620 02:39:30.648157 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:30.648161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:30.648267 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:30.648392 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:30.650424 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.651262 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.653546 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.653982 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.654106 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.657558 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:30.657564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:30.657567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:30.657612 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:30.660613 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:30.661419 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:30.661422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:30.661425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:30.661586 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:30.662041 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:30.662045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:30.662047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:30.662394 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:30.662623 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:30.663000 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 02:39:30.663202 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 02:39:30.668416 15972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:39:30.676024 15972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:39:30.707104 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.692225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.0.97.2:36877)
0620 02:39:30.693181 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:30.693194 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:30.693948 (+ 754us) server_negotiation.cc:407] Connection header received
0620 02:39:30.694362 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:30.694373 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:30.694687 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:30.694875 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:30.696089 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.698039 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.701936 (+ 3897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.704150 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.704421 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.704726 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:30.706117 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0620 02:39:30.706574 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":362,"threads_started":1}
I20260620 02:39:30.708418 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.692230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36877 (local address 127.0.0.1:33646)
0620 02:39:30.693804 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:30.693848 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:30.694104 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:30.695236 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:30.695248 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:30.695290 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:30.695816 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.695860 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.698337 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.698346 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.701608 (+ 3262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.701643 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.705360 (+ 3717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.705370 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.705725 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.705804 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 02:39:30.707509 (+ 1705us) client_negotiation.cc:240] Negotiation successful
0620 02:39:30.707906 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"thread_start_us":1163,"threads_started":1}
I20260620 02:39:30.714617 16230 tablet_service.cc:1467] Processing CreateTablet for tablet a6f8b4588e084a07851ff65e03ecc6cd (DEFAULT_TABLE table=test-table [id=028fbd9a43c249f39958e86d9cca3066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:30.716459 16230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f8b4588e084a07851ff65e03ecc6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:30.717016 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.694286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44267 (local address 127.0.0.1:40468)
0620 02:39:30.708740 (+ 14454us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:30.708764 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:30.708891 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:30.710251 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:30.710256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:30.710285 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:30.710534 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.710553 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.712463 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.712472 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.714143 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.714159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.716106 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.716114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.716335 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.716375 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:39:30.716464 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 02:39:30.716667 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14323,"thread_start_us":15252,"threads_started":1}
I20260620 02:39:30.717967 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.693897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.0.97.1:34435)
0620 02:39:30.706784 (+ 12887us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:30.706797 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:30.707960 (+ 1163us) server_negotiation.cc:407] Connection header received
0620 02:39:30.708084 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:30.708099 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:30.708416 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:30.708680 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:30.709394 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.711033 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.713859 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.715883 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.716032 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.716291 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:30.717420 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0620 02:39:30.717643 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12138,"thread_start_us":12437,"threads_started":1}
I20260620 02:39:30.718216 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.694913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.0.97.3:44267)
0620 02:39:30.709607 (+ 14694us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:30.709616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:30.709652 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:39:30.709763 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:30.709770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:30.710052 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:30.710269 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:30.710711 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.712272 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.714395 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.715872 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.717332 (+ 1460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.717582 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:30.717657 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 02:39:30.717833 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14464,"thread_start_us":14280,"threads_started":1}
I20260620 02:39:30.719386 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.693544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34435 (local address 127.0.0.1:43440)
0620 02:39:30.707636 (+ 14092us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:30.707658 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:30.707795 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:30.708920 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:30.708924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:30.708946 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:30.709228 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.709236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.711216 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.711222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.713609 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.713629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.716462 (+ 2833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.716469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.716676 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.716708 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:39:30.716798 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 02:39:30.716960 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13660,"mutex_wait_us":1,"thread_start_us":13480,"threads_started":1}
I20260620 02:39:30.722402 16364 tablet_service.cc:1467] Processing CreateTablet for tablet a6f8b4588e084a07851ff65e03ecc6cd (DEFAULT_TABLE table=test-table [id=028fbd9a43c249f39958e86d9cca3066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:30.724063 16364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f8b4588e084a07851ff65e03ecc6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:30.723559 16096 tablet_service.cc:1467] Processing CreateTablet for tablet a6f8b4588e084a07851ff65e03ecc6cd (DEFAULT_TABLE table=test-table [id=028fbd9a43c249f39958e86d9cca3066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:30.725327 16096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f8b4588e084a07851ff65e03ecc6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:30.739696 16448 tablet_bootstrap.cc:492] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3: Bootstrap starting.
I20260620 02:39:30.744287 16448 tablet_bootstrap.cc:654] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:30.745990 16448 log.cc:824] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:30.747486 16449 tablet_bootstrap.cc:492] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: Bootstrap starting.
I20260620 02:39:30.750670 16451 tablet_bootstrap.cc:492] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: Bootstrap starting.
I20260620 02:39:30.752025 16449 tablet_bootstrap.cc:654] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:30.753675 16449 log.cc:824] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:30.753777 16448 tablet_bootstrap.cc:492] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3: No bootstrap required, opened a new log
I20260620 02:39:30.754168 16448 ts_tablet_manager.cc:1397] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.002s
I20260620 02:39:30.755080 16451 tablet_bootstrap.cc:654] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:30.756736 16451 log.cc:824] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:30.757997 16449 tablet_bootstrap.cc:492] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: No bootstrap required, opened a new log
I20260620 02:39:30.758823 16449 ts_tablet_manager.cc:1397] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260620 02:39:30.762102 16451 tablet_bootstrap.cc:492] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: No bootstrap required, opened a new log
I20260620 02:39:30.762480 16451 ts_tablet_manager.cc:1397] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 02:39:30.764317 16448 raft_consensus.cc:348] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } } peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } }
I20260620 02:39:30.764776 16448 raft_consensus.cc:374] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:30.764889 16448 raft_consensus.cc:729] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884c4079aa784968bb1df80a930599c3, State: Initialized, Role: FOLLOWER
I20260620 02:39:30.765581 16448 consensus_queue.cc:260] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } } peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } }
I20260620 02:39:30.766865 16295 heartbeater.cc:503] Master 127.0.97.62:41621 was elected leader, sending a full tablet report...
I20260620 02:39:30.767671 16448 ts_tablet_manager.cc:1428] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260620 02:39:30.770941 16449 raft_consensus.cc:348] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } } peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } }
I20260620 02:39:30.771377 16449 raft_consensus.cc:374] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:30.771502 16449 raft_consensus.cc:729] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66cc8ab6b504b3da58ad372da5dccfd, State: Initialized, Role: FOLLOWER
I20260620 02:39:30.772286 16449 consensus_queue.cc:260] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } } peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } }
I20260620 02:39:30.773835 16429 heartbeater.cc:503] Master 127.0.97.62:41621 was elected leader, sending a full tablet report...
I20260620 02:39:30.774106 16451 raft_consensus.cc:348] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } } peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } }
I20260620 02:39:30.774657 16451 raft_consensus.cc:374] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:30.774770 16451 raft_consensus.cc:729] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141cb494da5b44cf89d2ba0d4402f7e1, State: Initialized, Role: FOLLOWER
I20260620 02:39:30.774772 16449 ts_tablet_manager.cc:1428] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260620 02:39:30.775597 16451 consensus_queue.cc:260] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } } peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } }
I20260620 02:39:30.777809 16451 ts_tablet_manager.cc:1428] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
W20260620 02:39:30.852735 16430 tablet.cc:2369] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:39:30.900398 16454 raft_consensus.cc:482] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:39:30.900739 16454 raft_consensus.cc:504] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } } peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } }
I20260620 02:39:30.902623 16454 leader_election.cc:288] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a66cc8ab6b504b3da58ad372da5dccfd (127.0.97.3:44267), 141cb494da5b44cf89d2ba0d4402f7e1 (127.0.97.1:34435)
I20260620 02:39:30.910889 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.903137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50557 (local address 127.0.97.1:34435)
0620 02:39:30.903425 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:30.903432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:30.904623 (+ 1191us) server_negotiation.cc:407] Connection header received
0620 02:39:30.904852 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:30.904863 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:30.904924 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:30.905030 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:30.905739 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.906477 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.908846 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.909663 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.909835 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.910015 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:30.910496 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 02:39:30.910644 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:39:30.911157 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.903208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49915 (local address 127.0.97.3:44267)
0620 02:39:30.903441 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:30.903509 (+ 68us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:30.904687 (+ 1178us) server_negotiation.cc:407] Connection header received
0620 02:39:30.904856 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:30.904859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:30.904919 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:30.905029 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:30.905618 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.906487 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.908839 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.909665 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.909852 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.910014 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:30.910765 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 02:39:30.910914 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:39:30.911396 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.903236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44267 (local address 127.0.97.2:49915)
0620 02:39:30.904536 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:30.904552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:30.904714 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:30.905044 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:30.905049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:30.905072 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:30.905453 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.905462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.906674 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.906677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.908649 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.908662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.910020 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.910028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.910235 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.910262 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 02:39:30.910389 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 02:39:30.910532 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"mutex_wait_us":133,"thread_start_us":763,"threads_started":1}
I20260620 02:39:30.911595 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.903237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34435 (local address 127.0.97.2:50557)
0620 02:39:30.904489 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:30.904506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:30.904688 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:30.905047 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:30.905051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:30.905106 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:30.905585 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.905598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.906663 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.906669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.908645 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:30.908662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.909976 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.909987 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:30.910243 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.910275 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:39:30.910447 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 02:39:30.910585 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"mutex_wait_us":87,"spinlock_wait_cycles":31616,"thread_start_us":868,"threads_started":1}
I20260620 02:39:30.912882 16384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f8b4588e084a07851ff65e03ecc6cd" candidate_uuid: "884c4079aa784968bb1df80a930599c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" is_pre_election: true
I20260620 02:39:30.912910 16116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f8b4588e084a07851ff65e03ecc6cd" candidate_uuid: "884c4079aa784968bb1df80a930599c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" is_pre_election: true
I20260620 02:39:30.913471 16384 raft_consensus.cc:2454] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 884c4079aa784968bb1df80a930599c3 in term 0.
I20260620 02:39:30.913559 16116 raft_consensus.cc:2454] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 884c4079aa784968bb1df80a930599c3 in term 0.
I20260620 02:39:30.914405 16183 leader_election.cc:302] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 884c4079aa784968bb1df80a930599c3, a66cc8ab6b504b3da58ad372da5dccfd; no voters:
I20260620 02:39:30.914883 16454 raft_consensus.cc:2781] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:39:30.915052 16454 raft_consensus.cc:482] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:39:30.915175 16454 raft_consensus.cc:3037] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:30.917550 16454 raft_consensus.cc:504] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } } peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } }
I20260620 02:39:30.918195 16454 leader_election.cc:288] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [CANDIDATE]: Term 1 election: Requested vote from peers a66cc8ab6b504b3da58ad372da5dccfd (127.0.97.3:44267), 141cb494da5b44cf89d2ba0d4402f7e1 (127.0.97.1:34435)
I20260620 02:39:30.918789 16116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f8b4588e084a07851ff65e03ecc6cd" candidate_uuid: "884c4079aa784968bb1df80a930599c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141cb494da5b44cf89d2ba0d4402f7e1"
I20260620 02:39:30.918809 16384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f8b4588e084a07851ff65e03ecc6cd" candidate_uuid: "884c4079aa784968bb1df80a930599c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66cc8ab6b504b3da58ad372da5dccfd"
I20260620 02:39:30.919018 16116 raft_consensus.cc:3037] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:30.919045 16384 raft_consensus.cc:3037] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:30.921449 16384 raft_consensus.cc:2454] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 884c4079aa784968bb1df80a930599c3 in term 1.
I20260620 02:39:30.921576 16116 raft_consensus.cc:2454] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 884c4079aa784968bb1df80a930599c3 in term 1.
I20260620 02:39:30.922041 16183 leader_election.cc:302] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 884c4079aa784968bb1df80a930599c3, a66cc8ab6b504b3da58ad372da5dccfd; no voters:
I20260620 02:39:30.922603 16454 raft_consensus.cc:2781] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:39:30.923233 16454 raft_consensus.cc:686] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 1 LEADER]: Becoming Leader. State: Replica: 884c4079aa784968bb1df80a930599c3, State: Running, Role: LEADER
I20260620 02:39:30.923808 16454 consensus_queue.cc:237] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } } peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } }
W20260620 02:39:30.927747 16162 tablet.cc:2369] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:39:30.929369 15971 catalog_manager.cc:5495] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 884c4079aa784968bb1df80a930599c3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "884c4079aa784968bb1df80a930599c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } health_report { overall_health: UNKNOWN } } }
I20260620 02:39:30.972019 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:30.965794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.0.97.2:36877)
0620 02:39:30.966043 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:30.966047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:30.966067 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:39:30.966325 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:30.966329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:30.966441 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:30.966576 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:39:30.966986 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.967724 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.969558 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:30.969980 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:30.970093 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:30.970676 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:39:30.971293 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:39:30.971418 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:30.971550 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 02:39:30.971630 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 02:39:30.981671 16384 raft_consensus.cc:1261] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd [term 1 FOLLOWER]: Refusing update from remote peer 884c4079aa784968bb1df80a930599c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:39:30.981673 16116 raft_consensus.cc:1261] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1 [term 1 FOLLOWER]: Refusing update from remote peer 884c4079aa784968bb1df80a930599c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:39:30.982856 16459 consensus_queue.cc:1035] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:39:30.983220 16454 consensus_queue.cc:1035] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:39:30.997160 16464 mvcc.cc:205] Tried to move back new op lower bound from 7298757308328669184 to 7298757308108394496. Current Snapshot: MvccSnapshot[applied={T|T < 7298757308328669184}]
I20260620 02:39:31.008859 15971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:39:31.009212 15971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:39:31.014798 15971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Sending DeleteTablet for 3 replicas of tablet a6f8b4588e084a07851ff65e03ecc6cd
I20260620 02:39:31.016947 16230 tablet_service.cc:1514] Processing DeleteTablet for tablet a6f8b4588e084a07851ff65e03ecc6cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33646
I20260620 02:39:31.017314 16364 tablet_service.cc:1514] Processing DeleteTablet for tablet a6f8b4588e084a07851ff65e03ecc6cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40468
I20260620 02:39:31.017858 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 16033
I20260620 02:39:31.017983 16096 tablet_service.cc:1514] Processing DeleteTablet for tablet a6f8b4588e084a07851ff65e03ecc6cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43440
I20260620 02:39:31.024474 16471 tablet_replica.cc:330] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3: stopping tablet replica
I20260620 02:39:31.025099 16471 raft_consensus.cc:2229] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:39:31.026023 16471 raft_consensus.cc:2258] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:31.038689 16471 ts_tablet_manager.cc:1905] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:39:31.042984 16473 tablet_replica.cc:330] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: stopping tablet replica
I20260620 02:39:31.044500 16472 tablet_replica.cc:330] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: stopping tablet replica
I20260620 02:39:31.045104 16472 raft_consensus.cc:2229] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:39:31.044195 16473 raft_consensus.cc:2229] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:39:31.045810 16473 raft_consensus.cc:2258] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:31.046326 16472 raft_consensus.cc:2258] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:31.048096 16471 ts_tablet_manager.cc:1918] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:31.048301 16471 log.cc:1196] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/a6f8b4588e084a07851ff65e03ecc6cd
I20260620 02:39:31.048955 16471 ts_tablet_manager.cc:1939] T a6f8b4588e084a07851ff65e03ecc6cd P 884c4079aa784968bb1df80a930599c3: Deleting consensus metadata
I20260620 02:39:31.050372 16473 ts_tablet_manager.cc:1905] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:39:31.050972 15956 catalog_manager.cc:4841] TS 884c4079aa784968bb1df80a930599c3 (127.0.97.2:36877): tablet a6f8b4588e084a07851ff65e03ecc6cd (table test-table [id=028fbd9a43c249f39958e86d9cca3066]) successfully deleted
I20260620 02:39:31.051911 16472 ts_tablet_manager.cc:1905] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:39:31 dist-test-slave-rpg6 krb5kdc[15927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:39:31.059253 16473 ts_tablet_manager.cc:1918] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:31.059485 16473 log.cc:1196] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/a6f8b4588e084a07851ff65e03ecc6cd
I20260620 02:39:31.060122 16473 ts_tablet_manager.cc:1939] T a6f8b4588e084a07851ff65e03ecc6cd P 141cb494da5b44cf89d2ba0d4402f7e1: Deleting consensus metadata
I20260620 02:39:31.061419 16472 ts_tablet_manager.cc:1918] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:31.061685 16472 log.cc:1196] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/a6f8b4588e084a07851ff65e03ecc6cd
I20260620 02:39:31.062295 16472 ts_tablet_manager.cc:1939] T a6f8b4588e084a07851ff65e03ecc6cd P a66cc8ab6b504b3da58ad372da5dccfd: Deleting consensus metadata
I20260620 02:39:31.062750 15959 catalog_manager.cc:4841] TS 141cb494da5b44cf89d2ba0d4402f7e1 (127.0.97.1:34435): tablet a6f8b4588e084a07851ff65e03ecc6cd (table test-table [id=028fbd9a43c249f39958e86d9cca3066]) successfully deleted
I20260620 02:39:31.064281 15956 catalog_manager.cc:4841] TS a66cc8ab6b504b3da58ad372da5dccfd (127.0.97.3:44267): tablet a6f8b4588e084a07851ff65e03ecc6cd (table test-table [id=028fbd9a43c249f39958e86d9cca3066]) successfully deleted
I20260620 02:39:31.066385 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:31.019087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.0.97.1:34435)
0620 02:39:31.019271 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:31.019278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:31.044113 (+ 24835us) server_negotiation.cc:407] Connection header received
0620 02:39:31.044392 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:31.044400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:31.044508 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:31.044630 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:31.046989 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:31.048334 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:31.050492 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:31.051152 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:31.051266 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:31.057520 (+ 6254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:31.057539 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:31.057560 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:31.057614 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:31.062687 (+ 5073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:31.063965 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:31.063975 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:31.063978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:31.064203 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:31.064749 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:31.064754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:31.064757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:31.065188 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:31.065443 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:31.065894 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 02:39:31.066107 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:39:31.067293 16156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:31.168052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 16033
I20260620 02:39:31.188095 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 16167
Jun 20 02:39:31 dist-test-slave-rpg6 krb5kdc[15927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:39:31.211119 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:31.189045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.0.97.2:36877)
0620 02:39:31.189235 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:31.189240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:31.189262 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:39:31.189419 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:31.189424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:31.189505 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:31.189612 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:31.190947 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:31.192175 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:31.193731 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:31.194435 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:31.194625 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:31.203727 (+ 9102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:31.203747 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:31.203764 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:31.203813 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:31.207405 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:31.208483 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:31.208498 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:31.208502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:31.208723 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:31.209194 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:31.209199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:31.209201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:31.209756 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:31.210222 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:31.210478 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 02:39:31.210717 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:39:31.212036 16290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:31.341912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 16167
I20260620 02:39:31.362293 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 16301
Jun 20 02:39:31 dist-test-slave-rpg6 krb5kdc[15927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:39:31.380584 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:31.363305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.0.97.3:44267)
0620 02:39:31.363532 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:31.363537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:31.363556 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:39:31.363614 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:31.363618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:31.363683 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:31.363928 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:31.365432 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:31.366572 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:31.368191 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:31.368630 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:31.368745 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:31.373893 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:31.373912 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:31.373930 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:31.373979 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:31.377251 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:31.378128 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:31.378137 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:31.378140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:31.378298 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:31.378678 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:31.378682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:31.378684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:31.379090 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:31.379375 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:31.379742 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 02:39:31.380094 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 02:39:31.381561 16424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:31.475618 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 16301
I20260620 02:39:31.496023 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 15940
I20260620 02:39:31.497363 16002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:31.613082 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 15940
I20260620 02:39:51.637768 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41621
--webserver_interface=127.0.97.62
--webserver_port=39503
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:39:52.021061 16479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:52.021432 16479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:52.021607 16479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:52.029626 16479 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:39:52.029767 16479 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:39:52.029808 16479 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:39:52.029894 16479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:39:52.029986 16479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:52.030092 16479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:52.030155 16479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:39:52.030249 16479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:39:52.041337 16479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:52.042793 16479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:52.044373 16479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:52.071851 16486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:52.071807 16485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:52.071725 16488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:52.072687 16479 server_base.cc:1034] running on GCE node
I20260620 02:39:52.073746 16479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:52.075799 16479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:52.077229 16479 hybrid_clock.cc:648] HybridClock initialized: now 1781923192077126 us; error 72 us; skew 500 ppm
I20260620 02:39:52.077718 16479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:52 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:52.083635 16479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:39:52.085883 16479 webserver.cc:460] Webserver started at http://127.0.97.62:39503/ using document root <none> and password file <none>
I20260620 02:39:52.086845 16479 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:52.087028 16479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:52.094853 16479 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:39:52.100490 16495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:52.103329 16479 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260620 02:39:52.103646 16479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "57cb051a75554f298208f7cea386dc10"
format_stamp: "Formatted at 2026-06-20 02:39:28 on dist-test-slave-rpg6"
server_key: "ff5f8c6ae0e6298008dc9a4e552e2640"
server_key_iv: "74001b4eebfab1588b0a4c93383cc08c"
server_key_version: "encryptionkey@0"
I20260620 02:39:52.105015 16479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:52.123139 16479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:52.129221 16479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:52.129653 16479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:52.153229 16479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41621
I20260620 02:39:52.153272 16546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41621 every 8 connection(s)
I20260620 02:39:52.155057 16479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:39:52.162606 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 16479
I20260620 02:39:52.163672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34435
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41621
--webserver_port=37463
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:39:52.166992 16547 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:39:52.172973 16547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Bootstrap starting.
I20260620 02:39:52.195454 16547 log.cc:824] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:52.218102 16547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:39:52.218822 16547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Bootstrap complete.
I20260620 02:39:52.232358 16547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:39:52.233352 16547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cb051a75554f298208f7cea386dc10, State: Initialized, Role: FOLLOWER
I20260620 02:39:52.234272 16547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:39:52.234552 16547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:39:52.234647 16547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:39:52.234838 16547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:39:52.239216 16547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:39:52.239876 16547 leader_election.cc:302] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57cb051a75554f298208f7cea386dc10; no voters:
I20260620 02:39:52.240651 16547 leader_election.cc:288] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:39:52.240798 16551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:39:52.242460 16551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 2 LEADER]: Becoming Leader. State: Replica: 57cb051a75554f298208f7cea386dc10, State: Running, Role: LEADER
I20260620 02:39:52.243589 16551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:39:52.244082 16547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:39:52.246672 16553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57cb051a75554f298208f7cea386dc10. Latest consensus state: current_term: 2 leader_uuid: "57cb051a75554f298208f7cea386dc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } } }
I20260620 02:39:52.246804 16552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "57cb051a75554f298208f7cea386dc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } } }
I20260620 02:39:52.247201 16553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:52.247301 16552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:39:52.248209 16558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:39:52.269335 16558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:39:52.271899 16558 catalog_manager.cc:1259] Loaded cluster ID: c83ec8c871464ffda9da705cbdba25c1
I20260620 02:39:52.272192 16558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:39:52.278316 16558 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:39:52.280975 16558 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Loaded TSK: 0
I20260620 02:39:52.322243 16558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Generated new TSK 1
I20260620 02:39:52.322728 16558 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:39:52.589181 16549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:52.589521 16549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:52.589623 16549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:52.597800 16549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:52.597950 16549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:52.598127 16549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:39:52.609674 16549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37463
--tserver_master_addrs=127.0.97.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:52.611447 16549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:52.613054 16549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:52.628185 16574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:52.628135 16575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:52.628595 16577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:52.630792 16549 server_base.cc:1034] running on GCE node
I20260620 02:39:52.631668 16549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:52.632918 16549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:52.634214 16549 hybrid_clock.cc:648] HybridClock initialized: now 1781923192634147 us; error 57 us; skew 500 ppm
I20260620 02:39:52.634640 16549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:52 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:52.639430 16549 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:39:52.642199 16549 webserver.cc:460] Webserver started at http://127.0.97.1:37463/ using document root <none> and password file <none>
I20260620 02:39:52.643285 16549 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:52.643566 16549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:52.652535 16549 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.007s
I20260620 02:39:52.655783 16584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:52.657476 16549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:39:52.657687 16549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "141cb494da5b44cf89d2ba0d4402f7e1"
format_stamp: "Formatted at 2026-06-20 02:39:29 on dist-test-slave-rpg6"
server_key: "0d40857a506fe6b5d01f5e7d3891cb45"
server_key_iv: "62cc683e3260cff4894f3df23f3b1317"
server_key_version: "encryptionkey@0"
I20260620 02:39:52.658762 16549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:52.680951 16549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:52.688025 16549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:52.688373 16549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:52.689600 16549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:52.691299 16549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:52.691394 16549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:52.691561 16549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:52.691658 16549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:52.735544 16549 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34435
I20260620 02:39:52.735659 16696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34435 every 8 connection(s)
I20260620 02:39:52.737553 16549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:39:52.738368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 16549
I20260620 02:39:52.739166 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36877
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41621
--webserver_port=35377
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:39:52 dist-test-slave-rpg6 krb5kdc[15927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:52.779512 16699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:52.742668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49969 (local address 127.0.97.62:41621)
0620 02:39:52.743506 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:52.743528 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:52.744766 (+ 1238us) server_negotiation.cc:407] Connection header received
0620 02:39:52.751583 (+ 6817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:52.751613 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:52.752257 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:52.752774 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:52.754726 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:52.756801 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:52.758754 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:52.759964 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:52.760202 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:52.766194 (+ 5992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:52.766212 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:52.766277 (+ 65us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:52.766420 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:52.772146 (+ 5726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:52.773422 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:52.773437 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:52.773449 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:52.773748 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:52.774651 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:52.774659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:52.774701 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:52.775724 (+ 1023us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:52.776093 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:52.777641 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0620 02:39:52.778111 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":396,"threads_started":1}
I20260620 02:39:52.779500 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:52.743084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:49969)
0620 02:39:52.744128 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:52.744177 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:52.750636 (+ 6459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:52.753051 (+ 2415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:52.753063 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:52.753757 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:52.754401 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:52.754424 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:52.757136 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:52.757140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:52.758530 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:52.758548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:52.760460 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:52.760469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:52.760992 (+ 523us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:52.762115 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:52.762160 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:52.765736 (+ 3576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:52.772596 (+ 6860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:52.772606 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:52.772621 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:52.773218 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:52.774049 (+ 831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:52.774056 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:52.774059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:52.774428 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:52.776040 (+ 1612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:52.776052 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:52.776876 (+ 824us) client_negotiation.cc:769] Sending connection context
0620 02:39:52.777328 (+ 452us) client_negotiation.cc:240] Negotiation successful
0620 02:39:52.778020 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":416,"threads_started":1}
I20260620 02:39:52.784721 16697 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41621
I20260620 02:39:52.785140 16697 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:52.786218 16697 heartbeater.cc:511] Master 127.0.97.62:41621 requested a full tablet report, sending...
I20260620 02:39:52.789269 16512 ts_manager.cc:194] Registered new tserver with Master: 141cb494da5b44cf89d2ba0d4402f7e1 (127.0.97.1:34435)
I20260620 02:39:52.791325 16512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49969
W20260620 02:39:53.157509 16701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:53.157922 16701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:53.158135 16701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:53.167755 16701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:53.168057 16701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:53.169090 16701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:39:53.180531 16701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35377
--tserver_master_addrs=127.0.97.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:53.182229 16701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:53.185034 16701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:53.205391 16706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:53.205475 16707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:53.205701 16709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:53.207571 16701 server_base.cc:1034] running on GCE node
I20260620 02:39:53.208459 16701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:53.210031 16701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:53.211493 16701 hybrid_clock.cc:648] HybridClock initialized: now 1781923193211395 us; error 87 us; skew 500 ppm
I20260620 02:39:53.211946 16701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:53 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:53.222206 16701 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:39:53.224462 16701 webserver.cc:460] Webserver started at http://127.0.97.2:35377/ using document root <none> and password file <none>
I20260620 02:39:53.225349 16701 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:53.225528 16701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:53.232494 16701 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:39:53.237088 16716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:53.238847 16701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:39:53.239200 16701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "884c4079aa784968bb1df80a930599c3"
format_stamp: "Formatted at 2026-06-20 02:39:29 on dist-test-slave-rpg6"
server_key: "631a7ddc9558ceff0527ac9b6e9a88ed"
server_key_iv: "647b0f46050141b0bf2de229ced4284d"
server_key_version: "encryptionkey@0"
I20260620 02:39:53.240410 16701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:53.274638 16701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:53.280583 16701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:53.280954 16701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:53.282271 16701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:53.284124 16701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:53.284224 16701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:53.284323 16701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:53.284384 16701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:53.332177 16701 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36877
I20260620 02:39:53.332291 16828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36877 every 8 connection(s)
I20260620 02:39:53.333796 16701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:39:53.341188 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 16701
I20260620 02:39:53.341948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44267
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41621
--webserver_port=36341
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:39:53 dist-test-slave-rpg6 krb5kdc[15927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:53.361548 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:53.336782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47747 (local address 127.0.97.62:41621)
0620 02:39:53.337651 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:53.337657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:53.338725 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 02:39:53.340833 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:53.340839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:53.340971 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:53.341186 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:53.345597 (+ 4411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.347177 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.350375 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.350906 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.351064 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:53.354919 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:53.354926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:53.354930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:53.354968 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:53.357800 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:53.358697 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:53.358703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:53.358705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:53.358839 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:53.359560 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:53.359565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:53.359567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:53.359944 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:53.360115 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:53.360878 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 02:39:53.361147 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":505,"threads_started":1}
I20260620 02:39:53.362663 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:53.337498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:47747)
0620 02:39:53.338462 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:53.338521 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:53.340473 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:53.341841 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:53.341857 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:53.343170 (+ 1313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:53.344911 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:53.344971 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.347508 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.347517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:53.350038 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:53.350081 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.351191 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.351200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:53.351471 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:53.352257 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:53.352287 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:53.354608 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:53.358000 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:53.358011 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:53.358033 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:53.358549 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:53.358960 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:53.358966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:53.358970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:53.359397 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:53.360079 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:53.360088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:53.360494 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 02:39:53.360782 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 02:39:53.361401 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":391,"threads_started":1}
I20260620 02:39:53.364845 16829 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41621
I20260620 02:39:53.365268 16829 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:53.366320 16829 heartbeater.cc:511] Master 127.0.97.62:41621 requested a full tablet report, sending...
I20260620 02:39:53.368489 16512 ts_manager.cc:194] Registered new tserver with Master: 884c4079aa784968bb1df80a930599c3 (127.0.97.2:36877)
I20260620 02:39:53.369681 16512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47747
W20260620 02:39:53.725742 16833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:39:53.726070 16833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:39:53.726171 16833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:39:53.733917 16833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:39:53.734063 16833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:39:53.734184 16833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:39:53.746227 16833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36341
--tserver_master_addrs=127.0.97.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:39:53.747673 16833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:39:53.749154 16833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:39:53.770330 16839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:53.770480 16841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:39:53.770324 16838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:39:53.771423 16833 server_base.cc:1034] running on GCE node
I20260620 02:39:53.772558 16833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:39:53.774287 16833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:39:53.775671 16833 hybrid_clock.cc:648] HybridClock initialized: now 1781923193775581 us; error 81 us; skew 500 ppm
I20260620 02:39:53.776306 16833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:39:53 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:39:53.780601 16833 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:39:53.782956 16833 webserver.cc:460] Webserver started at http://127.0.97.3:36341/ using document root <none> and password file <none>
I20260620 02:39:53.784022 16833 fs_manager.cc:359] Metadata directory not provided
I20260620 02:39:53.784400 16833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:39:53.791685 16833 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:39:53.795442 16848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:53.797135 16697 heartbeater.cc:503] Master 127.0.97.62:41621 was elected leader, sending a full tablet report...
I20260620 02:39:53.798144 16833 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:39:53.798674 16833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "a66cc8ab6b504b3da58ad372da5dccfd"
format_stamp: "Formatted at 2026-06-20 02:39:30 on dist-test-slave-rpg6"
server_key: "24756062d51eab65ecb5c8d5c2085709"
server_key_iv: "a779e60235270f2de648e795d4f75b42"
server_key_version: "encryptionkey@0"
I20260620 02:39:53.800074 16833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:39:53.817209 16833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:39:53.823765 16833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:39:53.824160 16833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:39:53.825461 16833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:39:53.827116 16833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:39:53.827222 16833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:53.827301 16833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:39:53.827356 16833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:39:53.873832 16833 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44267
I20260620 02:39:53.873889 16960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44267 every 8 connection(s)
I20260620 02:39:53.875559 16833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:39:53.883599 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 16833
Jun 20 02:39:53 dist-test-slave-rpg6 krb5kdc[15927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:39:53.910429 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:53.879176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50883 (local address 127.0.97.62:41621)
0620 02:39:53.879808 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:53.879818 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:53.882126 (+ 2308us) server_negotiation.cc:407] Connection header received
0620 02:39:53.884860 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:53.884874 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:53.885022 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:53.885140 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:53.887768 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.888769 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.891209 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.891967 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.892089 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:53.897481 (+ 5392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:53.897492 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:53.897498 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:53.897558 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:53.903631 (+ 6073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:53.905958 (+ 2327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:53.905963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:53.905966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:53.906149 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:53.907127 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:53.907133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:53.907135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:53.907643 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:53.907799 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:53.909634 (+ 1835us) server_negotiation.cc:299] Negotiation successful
0620 02:39:53.909891 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":479,"spinlock_wait_cycles":11136,"thread_start_us":359,"threads_started":1}
I20260620 02:39:53.912237 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:53.880419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:50883)
0620 02:39:53.881891 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:53.881950 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:53.884448 (+ 2498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:53.885491 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:53.885502 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:53.886547 (+ 1045us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:39:53.887559 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:53.887583 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.889298 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.889303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:53.890981 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:53.891005 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.892253 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.892260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:53.892510 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:53.893353 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:39:53.893382 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:39:53.897157 (+ 3775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:39:53.904351 (+ 7194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:53.904386 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:53.904413 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:53.905729 (+ 1316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:53.906589 (+ 860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:39:53.906598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:39:53.906603 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:39:53.906934 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:39:53.908085 (+ 1151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:39:53.908096 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:39:53.908549 (+ 453us) client_negotiation.cc:769] Sending connection context
0620 02:39:53.909490 (+ 941us) client_negotiation.cc:240] Negotiation successful
0620 02:39:53.910627 (+ 1137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"thread_start_us":816,"threads_started":1}
I20260620 02:39:53.912659 16966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:53.885678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.0.97.62:41621)
0620 02:39:53.888360 (+ 2682us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:53.888364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:53.888381 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:39:53.888425 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:53.888429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:53.888512 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:53.888620 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:53.890802 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.892018 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.894985 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.895505 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.895630 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:53.901531 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:53.901556 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:53.901566 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:53.901646 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:53.907291 (+ 5645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:53.909175 (+ 1884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:53.909183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:53.909187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:53.910022 (+ 835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:53.910898 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:53.910911 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:53.910914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:53.911278 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:53.911585 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:53.911974 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 02:39:53.912274 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2547,"thread_start_us":1634,"threads_started":1}
I20260620 02:39:53.914510 16961 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41621
I20260620 02:39:53.914917 16961 heartbeater.cc:464] Registering TS with master...
I20260620 02:39:53.915956 16961 heartbeater.cc:511] Master 127.0.97.62:41621 requested a full tablet report, sending...
I20260620 02:39:53.917737 16512 ts_manager.cc:194] Registered new tserver with Master: a66cc8ab6b504b3da58ad372da5dccfd (127.0.97.3:44267)
I20260620 02:39:53.918684 16512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50883
I20260620 02:39:53.932257 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:39:53.941325 16966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:53.933861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.0.97.62:41621)
0620 02:39:53.934110 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:53.934120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:53.934606 (+ 486us) server_negotiation.cc:407] Connection header received
0620 02:39:53.935014 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:53.935019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:53.935214 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:53.935576 (+ 362us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:39:53.936342 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.937342 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.939173 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.939673 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.939794 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:53.940509 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:39:53.940738 (+ 229us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 02:39:53.941092 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39668: token expired
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 02:39:53.941500 16966 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39668: token expired
W20260620 02:39:53.942652 16435 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:41621)
I20260620 02:39:53.960604 16966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:53.943246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.0.97.62:41621)
0620 02:39:53.943425 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:53.943429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:53.943446 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:39:53.943555 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:53.943559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:53.943621 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:53.943704 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:53.945707 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.946577 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.948253 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:53.948694 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:53.948845 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:53.951968 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:53.951975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:53.951978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:53.952022 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:53.955761 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:53.956785 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:53.956792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:53.956795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:53.956961 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:53.957368 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:53.957373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:53.957377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:53.957850 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:53.958418 (+ 568us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:53.959621 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0620 02:39:53.960210 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:39:53.965747 16512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:39:53.976398 16512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:39:54.027820 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:53.999294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.0.97.1:34435)
0620 02:39:54.013196 (+ 13902us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:54.013203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:54.013810 (+ 607us) server_negotiation.cc:407] Connection header received
0620 02:39:54.014108 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:54.014116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:54.014373 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:54.014833 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:54.016039 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.018686 (+ 2647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.024570 (+ 5884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.026351 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.026506 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.026801 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:54.027226 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 02:39:54.027435 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13738,"thread_start_us":13674,"threads_started":1}
I20260620 02:39:54.028363 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:53.999270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.0.97.3:44267)
0620 02:39:54.013144 (+ 13874us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:54.013155 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:54.013194 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:39:54.013302 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:54.013311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:54.013724 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:54.014115 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:54.015844 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.019119 (+ 3275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.024568 (+ 5449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.026351 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.026505 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.026800 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:54.027714 (+ 914us) server_negotiation.cc:299] Negotiation successful
0620 02:39:54.027979 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13675,"thread_start_us":13506,"threads_started":1}
I20260620 02:39:54.028400 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:53.998726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44267 (local address 127.0.0.1:50108)
0620 02:39:54.012240 (+ 13514us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.012299 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:54.012518 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:54.014738 (+ 2220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:54.014749 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:54.014792 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:54.015534 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:54.015550 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.019313 (+ 3763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.019318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:54.024329 (+ 5011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:54.024341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.026973 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.026979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:54.027153 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.027187 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:39:54.027277 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 02:39:54.027425 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13205,"mutex_wait_us":225,"thread_start_us":14536,"threads_started":1}
I20260620 02:39:54.029052 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:53.998890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34435 (local address 127.0.0.1:38646)
0620 02:39:54.013648 (+ 14758us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.013672 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:54.013808 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:54.015241 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:54.015249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:54.015273 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:54.015801 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:54.015811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.019272 (+ 3461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.019280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:54.024275 (+ 4995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:54.024300 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.026641 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.026648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:54.026877 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.026947 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 02:39:54.027114 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 02:39:54.027364 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14542,"mutex_wait_us":155,"thread_start_us":12840,"threads_started":1}
I20260620 02:39:54.030067 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.014176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.0.97.2:36877)
0620 02:39:54.017928 (+ 3752us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:54.017941 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:54.020289 (+ 2348us) server_negotiation.cc:407] Connection header received
0620 02:39:54.020606 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:54.020614 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:54.020877 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:54.021121 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:54.021953 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.023213 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.025861 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.027230 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.027363 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.027583 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:54.028522 (+ 939us) server_negotiation.cc:299] Negotiation successful
0620 02:39:54.029713 (+ 1191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3482,"thread_start_us":4159,"threads_started":1}
I20260620 02:39:54.030469 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.014111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36877 (local address 127.0.0.1:58814)
0620 02:39:54.020132 (+ 6021us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.020158 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:54.020358 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:54.021103 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:54.021108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:54.021141 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:54.021769 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:54.021780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.023417 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.023422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:54.025649 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:54.025666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.027669 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.027678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:54.027928 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.027963 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:39:54.028071 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 02:39:54.028239 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5123,"thread_start_us":4924,"threads_started":1}
I20260620 02:39:54.035220 16632 tablet_service.cc:1467] Processing CreateTablet for tablet 7e71d520a274411ea6c95e03b1c435d6 (DEFAULT_TABLE table=test-table [id=d1dd09a4f7aa4fb598bd4f12484982f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:54.035727 16764 tablet_service.cc:1467] Processing CreateTablet for tablet 7e71d520a274411ea6c95e03b1c435d6 (DEFAULT_TABLE table=test-table [id=d1dd09a4f7aa4fb598bd4f12484982f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:54.035470 16896 tablet_service.cc:1467] Processing CreateTablet for tablet 7e71d520a274411ea6c95e03b1c435d6 (DEFAULT_TABLE table=test-table [id=d1dd09a4f7aa4fb598bd4f12484982f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:39:54.037027 16632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e71d520a274411ea6c95e03b1c435d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:54.038034 16764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e71d520a274411ea6c95e03b1c435d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:54.038033 16896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e71d520a274411ea6c95e03b1c435d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:39:54.055255 16978 tablet_bootstrap.cc:492] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: Bootstrap starting.
I20260620 02:39:54.062094 16978 tablet_bootstrap.cc:654] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:54.063009 16980 tablet_bootstrap.cc:492] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: Bootstrap starting.
I20260620 02:39:54.064860 16978 log.cc:824] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:54.065003 16979 tablet_bootstrap.cc:492] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: Bootstrap starting.
I20260620 02:39:54.073148 16979 tablet_bootstrap.cc:654] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:54.074488 16979 log.cc:824] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:54.076151 16978 tablet_bootstrap.cc:492] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: No bootstrap required, opened a new log
I20260620 02:39:54.076517 16978 ts_tablet_manager.cc:1397] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.010s
I20260620 02:39:54.076787 16980 tablet_bootstrap.cc:654] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: Neither blocks nor log segments found. Creating new log.
I20260620 02:39:54.077668 16979 tablet_bootstrap.cc:492] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: No bootstrap required, opened a new log
I20260620 02:39:54.077965 16979 ts_tablet_manager.cc:1397] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260620 02:39:54.078249 16980 log.cc:824] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: Log is configured to *not* fsync() on all Append() calls
I20260620 02:39:54.082357 16980 tablet_bootstrap.cc:492] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: No bootstrap required, opened a new log
I20260620 02:39:54.083331 16980 ts_tablet_manager.cc:1397] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.000s
I20260620 02:39:54.088932 16979 raft_consensus.cc:348] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } } peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } }
I20260620 02:39:54.089455 16979 raft_consensus.cc:374] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:54.089562 16979 raft_consensus.cc:729] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884c4079aa784968bb1df80a930599c3, State: Initialized, Role: FOLLOWER
I20260620 02:39:54.089713 16978 raft_consensus.cc:348] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } } peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } }
I20260620 02:39:54.090197 16978 raft_consensus.cc:374] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:54.090301 16978 raft_consensus.cc:729] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141cb494da5b44cf89d2ba0d4402f7e1, State: Initialized, Role: FOLLOWER
I20260620 02:39:54.090329 16979 consensus_queue.cc:260] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } } peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } }
I20260620 02:39:54.090926 16978 consensus_queue.cc:260] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } } peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } }
I20260620 02:39:54.091940 16829 heartbeater.cc:503] Master 127.0.97.62:41621 was elected leader, sending a full tablet report...
I20260620 02:39:54.092598 16979 ts_tablet_manager.cc:1428] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260620 02:39:54.097893 16978 ts_tablet_manager.cc:1428] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: Time spent starting tablet: real 0.021s user 0.011s sys 0.010s
I20260620 02:39:54.103015 16980 raft_consensus.cc:348] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } } peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } }
I20260620 02:39:54.103888 16980 raft_consensus.cc:374] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:39:54.104004 16980 raft_consensus.cc:729] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66cc8ab6b504b3da58ad372da5dccfd, State: Initialized, Role: FOLLOWER
I20260620 02:39:54.104785 16980 consensus_queue.cc:260] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } } peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } }
I20260620 02:39:54.107999 16961 heartbeater.cc:503] Master 127.0.97.62:41621 was elected leader, sending a full tablet report...
I20260620 02:39:54.109208 16980 ts_tablet_manager.cc:1428] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: Time spent starting tablet: real 0.025s user 0.016s sys 0.010s
W20260620 02:39:54.127795 16962 tablet.cc:2369] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:39:54.217083 16985 raft_consensus.cc:482] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:39:54.217425 16985 raft_consensus.cc:504] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } } peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } }
I20260620 02:39:54.219794 16985 leader_election.cc:288] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a66cc8ab6b504b3da58ad372da5dccfd (127.0.97.3:44267), 884c4079aa784968bb1df80a930599c3 (127.0.97.2:36877)
I20260620 02:39:54.229753 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.220846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42879 (local address 127.0.97.3:44267)
0620 02:39:54.221245 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:54.221250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:54.221671 (+ 421us) server_negotiation.cc:407] Connection header received
0620 02:39:54.221920 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:54.221925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:54.222016 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:54.222133 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:54.223036 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.224122 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.226648 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.228025 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.228142 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.228275 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:54.229384 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0620 02:39:54.229521 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260620 02:39:54.229974 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.220675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44267 (local address 127.0.97.1:42879)
0620 02:39:54.221522 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.221546 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:54.221709 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:54.222374 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:54.222379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:54.222407 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:54.222828 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:54.222837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.224256 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.224262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:54.226428 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:54.226443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.228653 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.228663 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:54.228914 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.228971 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 02:39:54.229281 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 02:39:54.229497 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":475,"threads_started":1}
I20260620 02:39:54.232652 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.221734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36877 (local address 127.0.97.1:59095)
0620 02:39:54.222812 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.222832 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:39:54.223040 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:39:54.223450 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:39:54.223455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:39:54.223483 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:39:54.223879 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:54.223891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.225411 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.225419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:54.227753 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:39:54.227770 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.230772 (+ 3002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.230782 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:39:54.231045 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.231098 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 02:39:54.231352 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 02:39:54.231618 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":502,"threads_started":1}
I20260620 02:39:54.232409 16916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e71d520a274411ea6c95e03b1c435d6" candidate_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" is_pre_election: true
I20260620 02:39:54.233219 16916 raft_consensus.cc:2454] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 141cb494da5b44cf89d2ba0d4402f7e1 in term 0.
I20260620 02:39:54.233651 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.222010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59095 (local address 127.0.97.2:36877)
0620 02:39:54.222301 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:54.222307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:54.222979 (+ 672us) server_negotiation.cc:407] Connection header received
0620 02:39:54.223229 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:54.223238 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:54.223304 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:54.223410 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:39:54.224039 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.225187 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.228564 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.230188 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.230422 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.230548 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:54.233180 (+ 2632us) server_negotiation.cc:299] Negotiation successful
0620 02:39:54.233395 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 02:39:54.234218 16585 leader_election.cc:302] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141cb494da5b44cf89d2ba0d4402f7e1, a66cc8ab6b504b3da58ad372da5dccfd; no voters:
I20260620 02:39:54.234930 16985 raft_consensus.cc:2781] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:39:54.234941 16784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e71d520a274411ea6c95e03b1c435d6" candidate_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884c4079aa784968bb1df80a930599c3" is_pre_election: true
I20260620 02:39:54.235230 16985 raft_consensus.cc:482] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:39:54.235370 16985 raft_consensus.cc:3037] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:54.235492 16784 raft_consensus.cc:2454] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 141cb494da5b44cf89d2ba0d4402f7e1 in term 0.
I20260620 02:39:54.238111 16985 raft_consensus.cc:504] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } } peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } }
I20260620 02:39:54.238852 16985 leader_election.cc:288] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [CANDIDATE]: Term 1 election: Requested vote from peers a66cc8ab6b504b3da58ad372da5dccfd (127.0.97.3:44267), 884c4079aa784968bb1df80a930599c3 (127.0.97.2:36877)
I20260620 02:39:54.239679 16916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e71d520a274411ea6c95e03b1c435d6" candidate_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66cc8ab6b504b3da58ad372da5dccfd"
I20260620 02:39:54.239959 16916 raft_consensus.cc:3037] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:39:54.240057 16784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e71d520a274411ea6c95e03b1c435d6" candidate_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884c4079aa784968bb1df80a930599c3"
I20260620 02:39:54.240438 16784 raft_consensus.cc:3037] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3 [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:39:54.241106 16698 tablet.cc:2369] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:39:54.242683 16916 raft_consensus.cc:2454] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 141cb494da5b44cf89d2ba0d4402f7e1 in term 1.
I20260620 02:39:54.243196 16585 leader_election.cc:302] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 141cb494da5b44cf89d2ba0d4402f7e1, a66cc8ab6b504b3da58ad372da5dccfd; no voters:
I20260620 02:39:54.243530 16985 raft_consensus.cc:2781] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:39:54.244230 16985 raft_consensus.cc:686] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 1 LEADER]: Becoming Leader. State: Replica: 141cb494da5b44cf89d2ba0d4402f7e1, State: Running, Role: LEADER
I20260620 02:39:54.244940 16985 consensus_queue.cc:237] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } } peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } }
I20260620 02:39:54.247845 16784 raft_consensus.cc:2454] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 141cb494da5b44cf89d2ba0d4402f7e1 in term 1.
I20260620 02:39:54.250803 16512 catalog_manager.cc:5495] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 141cb494da5b44cf89d2ba0d4402f7e1 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "141cb494da5b44cf89d2ba0d4402f7e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 } health_report { overall_health: UNKNOWN } } }
I20260620 02:39:54.321734 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.314701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.0.97.1:34435)
0620 02:39:54.314997 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:54.315005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:54.315129 (+ 124us) server_negotiation.cc:407] Connection header received
0620 02:39:54.315408 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:54.315412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:54.315499 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:54.315679 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:39:54.316091 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.316978 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.319177 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.319708 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.319887 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.320512 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:39:54.321058 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:39:54.321228 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:54.321318 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 02:39:54.321443 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260620 02:39:54.334023 16784 raft_consensus.cc:1261] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3 [term 1 FOLLOWER]: Refusing update from remote peer 141cb494da5b44cf89d2ba0d4402f7e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:39:54.334177 16916 raft_consensus.cc:1261] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd [term 1 FOLLOWER]: Refusing update from remote peer 141cb494da5b44cf89d2ba0d4402f7e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:39:54.335282 16985 consensus_queue.cc:1035] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66cc8ab6b504b3da58ad372da5dccfd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:39:54.335667 16989 consensus_queue.cc:1035] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "884c4079aa784968bb1df80a930599c3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260620 02:39:54.336634 16830 tablet.cc:2369] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:39:54.354539 16992 mvcc.cc:205] Tried to move back new op lower bound from 7298757403976368128 to 7298757403632230400. Current Snapshot: MvccSnapshot[applied={T|T < 7298757403976368128}]
I20260620 02:39:54.374612 16510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:39:54.375342 16510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:39:54.379679 16510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Sending DeleteTablet for 3 replicas of tablet 7e71d520a274411ea6c95e03b1c435d6
I20260620 02:39:54.382136 16764 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e71d520a274411ea6c95e03b1c435d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58814
I20260620 02:39:54.381975 16896 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e71d520a274411ea6c95e03b1c435d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50108
I20260620 02:39:54.381982 16632 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e71d520a274411ea6c95e03b1c435d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:39:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38646
I20260620 02:39:54.382958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 16549
I20260620 02:39:54.396412 17003 tablet_replica.cc:330] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: stopping tablet replica
I20260620 02:39:54.397476 17002 tablet_replica.cc:330] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: stopping tablet replica
I20260620 02:39:54.397994 17002 raft_consensus.cc:2229] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:39:54.398715 17002 raft_consensus.cc:2258] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:54.399717 17001 tablet_replica.cc:330] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: stopping tablet replica
I20260620 02:39:54.400177 17001 raft_consensus.cc:2229] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:39:54.400185 17003 raft_consensus.cc:2229] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:39:54.400887 17003 raft_consensus.cc:2258] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:54.400895 17001 raft_consensus.cc:2258] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:39:54.403505 17003 ts_tablet_manager.cc:1905] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:39:54.403509 17001 ts_tablet_manager.cc:1905] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:39:54.406430 17002 ts_tablet_manager.cc:1905] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:39:54.414683 17003 ts_tablet_manager.cc:1918] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:54.415020 17003 log.cc:1196] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/7e71d520a274411ea6c95e03b1c435d6
I20260620 02:39:54.415737 17003 ts_tablet_manager.cc:1939] T 7e71d520a274411ea6c95e03b1c435d6 P 884c4079aa784968bb1df80a930599c3: Deleting consensus metadata
I20260620 02:39:54.417136 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.384005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.0.97.1:34435)
0620 02:39:54.384248 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:54.384254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:54.384283 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:39:54.384372 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:54.384381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:54.384486 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:54.384641 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:54.387364 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.389240 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.395079 (+ 5839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.396476 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.396683 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.403764 (+ 7081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:54.403788 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:54.403808 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:54.403900 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:54.410105 (+ 6205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:54.412261 (+ 2156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:54.412275 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:54.412279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:54.412777 (+ 498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:54.413904 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:54.413912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:54.413916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:54.414555 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:54.415961 (+ 1406us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:54.416213 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 02:39:54.416648 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:39:54.418206 16496 catalog_manager.cc:4841] TS 884c4079aa784968bb1df80a930599c3 (127.0.97.2:36877): tablet 7e71d520a274411ea6c95e03b1c435d6 (table test-table [id=d1dd09a4f7aa4fb598bd4f12484982f8]) successfully deleted
I20260620 02:39:54.419528 17001 ts_tablet_manager.cc:1918] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:54.419617 16692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:54.419934 17001 log.cc:1196] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/7e71d520a274411ea6c95e03b1c435d6
I20260620 02:39:54.425490 17002 ts_tablet_manager.cc:1918] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:39:54.425738 17002 log.cc:1196] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/7e71d520a274411ea6c95e03b1c435d6
I20260620 02:39:54.426326 17002 ts_tablet_manager.cc:1939] T 7e71d520a274411ea6c95e03b1c435d6 P a66cc8ab6b504b3da58ad372da5dccfd: Deleting consensus metadata
I20260620 02:39:54.428489 16496 catalog_manager.cc:4841] TS a66cc8ab6b504b3da58ad372da5dccfd (127.0.97.3:44267): tablet 7e71d520a274411ea6c95e03b1c435d6 (table test-table [id=d1dd09a4f7aa4fb598bd4f12484982f8]) successfully deleted
I20260620 02:39:54.532089 17001 ts_tablet_manager.cc:1939] T 7e71d520a274411ea6c95e03b1c435d6 P 141cb494da5b44cf89d2ba0d4402f7e1: Deleting consensus metadata
I20260620 02:39:54.533007 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 16549
I20260620 02:39:54.553447 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 16701
W20260620 02:39:54.553493 16499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34435 (error 108)
I20260620 02:39:54.555413 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.554724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34435 (local address 127.0.0.1:38672)
0620 02:39:54.555142 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.555238 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":222,"threads_started":1}
W20260620 02:39:54.555644 16499 catalog_manager.cc:4584] TS 141cb494da5b44cf89d2ba0d4402f7e1 (127.0.97.1:34435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e71d520a274411ea6c95e03b1c435d6: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34435: connect: Connection refused (error 111)
I20260620 02:39:54.569157 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.554274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.0.97.2:36877)
0620 02:39:54.554468 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:54.554475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:54.554549 (+ 74us) server_negotiation.cc:407] Connection header received
0620 02:39:54.554761 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:54.554767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:54.554865 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:54.554977 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:54.556546 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.557304 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.558615 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.559021 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.559161 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.562260 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:54.562277 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:54.562293 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:54.562333 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:54.565939 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:54.566807 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:54.566815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:54.566818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:54.566952 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:54.567359 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:54.567382 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:54.567388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:54.567811 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:54.568245 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:54.568558 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 02:39:54.568801 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:39:54.569877 16824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:54.609676 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.609093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34435 (local address 127.0.0.1:38688)
0620 02:39:54.609355 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.609454 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 02:39:54.610929 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.610410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34435 (local address 127.0.0.1:38702)
0620 02:39:54.610676 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.610748 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260620 02:39:54.672139 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.671549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34435 (local address 127.0.0.1:38718)
0620 02:39:54.671818 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.671972 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 02:39:54.673731 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.673130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34435 (local address 127.0.0.1:38720)
0620 02:39:54.673354 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.673456 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 02:39:54.674715 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 16701
I20260620 02:39:54.693312 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 16833
I20260620 02:39:54.715938 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.694768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.0.97.3:44267)
0620 02:39:54.695033 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 02:39:54.695038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:39:54.695097 (+ 59us) server_negotiation.cc:407] Connection header received
0620 02:39:54.695163 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:39:54.695167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:39:54.695255 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:39:54.695376 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:39:54.697406 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.698768 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.701414 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:39:54.701889 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:39:54.702066 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:39:54.706609 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:39:54.706627 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:39:54.706645 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:39:54.706690 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:39:54.711561 (+ 4871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:54.712715 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:54.712725 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:54.712727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:54.712899 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:39:54.713394 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:39:54.713398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:39:54.713400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:39:54.714040 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:39:54.714307 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 02:39:54.715131 (+ 824us) server_negotiation.cc:299] Negotiation successful
0620 02:39:54.715442 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 02:39:54.717152 16956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:54.754936 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.754422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34435 (local address 127.0.0.1:38736)
0620 02:39:54.754670 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.754778 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 02:39:54.756260 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.755794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34435 (local address 127.0.0.1:38742)
0620 02:39:54.756048 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.756121 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260620 02:39:54.817926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 16833
I20260620 02:39:54.836395 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 16479
I20260620 02:39:54.837659 16542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:39:54.961589 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:39:54.960797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34435 (local address 127.0.0.1:38746)
0620 02:39:54.961049 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 02:39:54.961261 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 02:39:54.961656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 16479
I20260620 02:40:14.988545 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41621
--webserver_interface=127.0.97.62
--webserver_port=39503
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:40:15.502977 17010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:15.503387 17010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:15.503563 17010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:15.513057 17010 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:40:15.513263 17010 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:40:15.513381 17010 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:40:15.513485 17010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:40:15.513571 17010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:15.513674 17010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:15.513768 17010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:40:15.513862 17010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:40:15.526757 17010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:15.528339 17010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:15.529925 17010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:15.547935 17015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:15.548069 17018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:15.548084 17016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:15.548403 17010 server_base.cc:1034] running on GCE node
I20260620 02:40:15.550299 17010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:15.552491 17010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:15.554211 17010 hybrid_clock.cc:648] HybridClock initialized: now 1781923215553970 us; error 126 us; skew 500 ppm
I20260620 02:40:15.554855 17010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:15 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:15.573982 17010 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:40:15.576699 17010 webserver.cc:460] Webserver started at http://127.0.97.62:39503/ using document root <none> and password file <none>
I20260620 02:40:15.577630 17010 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:15.577791 17010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:15.585969 17010 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260620 02:40:15.591286 17025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:15.594493 17010 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 02:40:15.594805 17010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "57cb051a75554f298208f7cea386dc10"
format_stamp: "Formatted at 2026-06-20 02:39:28 on dist-test-slave-rpg6"
server_key: "ff5f8c6ae0e6298008dc9a4e552e2640"
server_key_iv: "74001b4eebfab1588b0a4c93383cc08c"
server_key_version: "encryptionkey@0"
I20260620 02:40:15.596061 17010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:15.630429 17010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:15.636030 17010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:15.636371 17010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:15.666004 17010 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41621
I20260620 02:40:15.666119 17076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41621 every 8 connection(s)
I20260620 02:40:15.668331 17010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:40:15.676955 17077 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:40:15.678543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 17010
I20260620 02:40:15.679629 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34435
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41621
--webserver_port=37463
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:40:15.681516 17077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Bootstrap starting.
I20260620 02:40:15.718670 17077 log.cc:824] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:15.770409 17077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:40:15.771157 17077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Bootstrap complete.
I20260620 02:40:15.786859 17077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:40:15.788077 17077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cb051a75554f298208f7cea386dc10, State: Initialized, Role: FOLLOWER
I20260620 02:40:15.788890 17077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:40:15.789182 17077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:40:15.789307 17077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:40:15.789518 17077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:40:15.798270 17077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:40:15.799089 17077 leader_election.cc:302] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57cb051a75554f298208f7cea386dc10; no voters:
I20260620 02:40:15.800647 17077 leader_election.cc:288] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 02:40:15.800803 17081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 02:40:15.803215 17081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [term 3 LEADER]: Becoming Leader. State: Replica: 57cb051a75554f298208f7cea386dc10, State: Running, Role: LEADER
I20260620 02:40:15.804543 17077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:40:15.804647 17081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } }
I20260620 02:40:15.814759 17083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57cb051a75554f298208f7cea386dc10. Latest consensus state: current_term: 3 leader_uuid: "57cb051a75554f298208f7cea386dc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } } }
I20260620 02:40:15.816274 17083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:15.815274 17082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "57cb051a75554f298208f7cea386dc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57cb051a75554f298208f7cea386dc10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41621 } } }
I20260620 02:40:15.817117 17082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10 [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:15.830964 17088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:40:15.843072 17088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:40:15.845158 17088 catalog_manager.cc:1259] Loaded cluster ID: c83ec8c871464ffda9da705cbdba25c1
I20260620 02:40:15.845433 17088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:40:15.851373 17088 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:40:15.856693 17088 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Loaded TSK: 0
I20260620 02:40:15.857257 17088 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57cb051a75554f298208f7cea386dc10: Loaded TSK: 1
F20260620 02:40:15.859669 17088 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f21dc9459ad google::LogMessage::Fail() at ??:0
@ 0x7f21dc94888a google::LogMessage::SendToLog() at ??:0
@ 0x7f21dc9454de google::LogMessage::Flush() at ??:0
@ 0x7f21dc946a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f21e06a97d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f21e069873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f21f23781c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f21f2377783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f21f24032a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f21f24030c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f21f244d5e9 std::function<>::operator()() at ??:0
@ 0x7f21f237bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f21f2379c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f21f2401610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f21f2401439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f21df591345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f21df59491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f21df594799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f21df56feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f21e28496db start_thread at ??:0
@ 0x7f21db7df61f clone at ??:0
W20260620 02:40:16.346518 17079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:16.346870 17079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:16.347043 17079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:16.359251 17079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:16.359519 17079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:16.359818 17079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:40:16.371320 17079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37463
--tserver_master_addrs=127.0.97.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:16.372754 17079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:16.374167 17079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:16.398751 17127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:16.398953 17124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:16.398777 17125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:16.401096 17079 server_base.cc:1034] running on GCE node
I20260620 02:40:16.401995 17079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:16.403285 17079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:16.406805 17079 hybrid_clock.cc:648] HybridClock initialized: now 1781923216406728 us; error 37 us; skew 500 ppm
I20260620 02:40:16.407588 17079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:16 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:16.414438 17079 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:40:16.417038 17079 webserver.cc:460] Webserver started at http://127.0.97.1:37463/ using document root <none> and password file <none>
I20260620 02:40:16.418133 17079 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:16.418335 17079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:16.428815 17079 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 02:40:16.433315 17134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:16.436439 17079 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:40:16.436959 17079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "141cb494da5b44cf89d2ba0d4402f7e1"
format_stamp: "Formatted at 2026-06-20 02:39:29 on dist-test-slave-rpg6"
server_key: "0d40857a506fe6b5d01f5e7d3891cb45"
server_key_iv: "62cc683e3260cff4894f3df23f3b1317"
server_key_version: "encryptionkey@0"
I20260620 02:40:16.438665 17079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:16.463227 17079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:16.468523 17079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:16.468840 17079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:16.470077 17079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:16.471751 17079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:16.471925 17079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 02:40:16.472071 17079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:16.472169 17079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:16.530021 17079 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34435
I20260620 02:40:16.530424 17246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34435 every 8 connection(s)
I20260620 02:40:16.532544 17079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:40:16.540195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 17079
I20260620 02:40:16.541121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36877
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41621
--webserver_port=35377
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:40:16.541712 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:16.538412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:38875)
0620 02:40:16.539735 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:16.540130 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":510,"threads_started":1}
W20260620 02:40:16.543156 17247 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41621 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41621: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
I20260620 02:40:16.546136 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:16.545546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:45291)
0620 02:40:16.545790 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:16.545921 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 02:40:16.547538 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:16.547024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:35237)
0620 02:40:16.547218 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:16.547342 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 02:40:16.547986 17247 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:40:16.998634 17250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:16.998968 17250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:16.999128 17250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:17.007966 17250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:17.008149 17250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:17.008354 17250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:40:17.022423 17250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35377
--tserver_master_addrs=127.0.97.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:17.024133 17250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:17.025698 17250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:17.045949 17255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:17.046130 17256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:17.046274 17258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:17.047472 17250 server_base.cc:1034] running on GCE node
I20260620 02:40:17.049137 17250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:17.050976 17250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:17.054538 17250 hybrid_clock.cc:648] HybridClock initialized: now 1781923217054333 us; error 72 us; skew 500 ppm
I20260620 02:40:17.055006 17250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:17 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:17.060179 17250 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:40:17.063779 17250 webserver.cc:460] Webserver started at http://127.0.97.2:35377/ using document root <none> and password file <none>
I20260620 02:40:17.064776 17250 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:17.064939 17250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:17.073158 17250 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:40:17.078452 17265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:17.082964 17250 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.002s
I20260620 02:40:17.083750 17250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "884c4079aa784968bb1df80a930599c3"
format_stamp: "Formatted at 2026-06-20 02:39:29 on dist-test-slave-rpg6"
server_key: "631a7ddc9558ceff0527ac9b6e9a88ed"
server_key_iv: "647b0f46050141b0bf2de229ced4284d"
server_key_version: "encryptionkey@0"
I20260620 02:40:17.085732 17250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:17.147323 17250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:17.154233 17250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:17.154636 17250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:17.156425 17250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:17.159332 17250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:17.159471 17250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:17.160169 17250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:17.160426 17250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:17.230402 17250 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36877
I20260620 02:40:17.230959 17377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36877 every 8 connection(s)
I20260620 02:40:17.235463 17250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:40:17.244087 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:17.240255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:44599)
0620 02:40:17.242008 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:17.242366 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1170,"thread_start_us":805,"threads_started":1}
W20260620 02:40:17.245201 17378 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41621 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41621: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
I20260620 02:40:17.245386 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 17250
I20260620 02:40:17.246416 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:17.245929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:43207)
0620 02:40:17.246150 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:17.246251 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 02:40:17.246814 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44267
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41621
--webserver_port=36341
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:40:17.248504 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:17.247697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:55103)
0620 02:40:17.248015 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:17.248265 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260620 02:40:17.249217 17378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:40:17.550321 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:17.549262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:42331)
0620 02:40:17.549905 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:17.550060 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":350,"threads_started":1}
W20260620 02:40:17.739607 17381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:17.740049 17381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:17.740252 17381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:17.749204 17381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:17.749364 17381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:17.749550 17381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:40:17.760267 17381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36341
--tserver_master_addrs=127.0.97.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:17.761865 17381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:17.763455 17381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:17.781736 17387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:17.781682 17388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:17.781731 17390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:17.787411 17381 server_base.cc:1034] running on GCE node
I20260620 02:40:17.788277 17381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:17.789776 17381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:17.793434 17381 hybrid_clock.cc:648] HybridClock initialized: now 1781923217793252 us; error 134 us; skew 500 ppm
I20260620 02:40:17.794018 17381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:17 dist-test-slave-rpg6 krb5kdc[15927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:17.799062 17381 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:40:17.802001 17381 webserver.cc:460] Webserver started at http://127.0.97.3:36341/ using document root <none> and password file <none>
I20260620 02:40:17.803118 17381 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:17.803282 17381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:17.812193 17381 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:40:17.816414 17397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:17.818423 17381 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260620 02:40:17.818708 17381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "a66cc8ab6b504b3da58ad372da5dccfd"
format_stamp: "Formatted at 2026-06-20 02:39:30 on dist-test-slave-rpg6"
server_key: "24756062d51eab65ecb5c8d5c2085709"
server_key_iv: "a779e60235270f2de648e795d4f75b42"
server_key_version: "encryptionkey@0"
I20260620 02:40:17.821033 17381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:17.850823 17381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:17.855999 17381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:17.856289 17381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:17.857774 17381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:17.859424 17381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:17.859531 17381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:17.859652 17381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:17.859741 17381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:17.920789 17381 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44267
I20260620 02:40:17.920811 17509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44267 every 8 connection(s)
I20260620 02:40:17.922695 17381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:40:17.929226 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 17381
I20260620 02:40:17.932336 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:17.929304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:39463)
0620 02:40:17.930796 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:17.931139 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":619,"threads_started":1}
W20260620 02:40:17.933652 17510 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41621 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41621: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
I20260620 02:40:17.935009 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:17.934547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:47343)
0620 02:40:17.934757 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:17.934845 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 02:40:17.937198 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:17.936413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:49117)
0620 02:40:17.936803 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:17.936944 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
W20260620 02:40:17.938004 17510 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:40:18.251382 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:18.250056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:40697)
0620 02:40:18.250961 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:18.251086 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":407,"threads_started":1}
I20260620 02:40:18.553252 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:18.552349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:53563)
0620 02:40:18.552945 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:18.553055 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":301,"threads_started":1}
I20260620 02:40:18.940629 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:18.939195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:51439)
0620 02:40:18.940226 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:18.940361 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":369,"threads_started":1}
I20260620 02:40:19.253895 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:19.253014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:52873)
0620 02:40:19.253589 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:19.253698 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":335,"threads_started":1}
I20260620 02:40:19.555650 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:19.554653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:54135)
0620 02:40:19.555269 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:19.555370 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":376,"threads_started":1}
I20260620 02:40:19.944574 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:19.942762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:46645)
0620 02:40:19.944175 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:19.944371 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1270,"thread_start_us":1036,"threads_started":1}
I20260620 02:40:20.257109 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:20.255920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:34685)
0620 02:40:20.256728 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:20.256864 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":334,"threads_started":1}
I20260620 02:40:20.558163 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:20.557305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:58887)
0620 02:40:20.557891 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:20.557983 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":348,"threads_started":1}
I20260620 02:40:20.947417 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:20.946166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:58887)
0620 02:40:20.947091 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:20.947190 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":373,"threads_started":1}
I20260620 02:40:21.259497 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:21.258435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:36141)
0620 02:40:21.259050 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:21.259158 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":328,"threads_started":1}
I20260620 02:40:21.561504 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:21.559924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:59275)
0620 02:40:21.560864 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:21.560991 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":365,"threads_started":1}
I20260620 02:40:21.949908 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:21.949035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:34051)
0620 02:40:21.949623 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:21.949711 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":339,"threads_started":1}
I20260620 02:40:22.262051 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:22.261094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:42401)
0620 02:40:22.261732 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:22.261824 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":386,"threads_started":1}
I20260620 02:40:22.564090 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:22.562722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:44181)
0620 02:40:22.563644 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:22.563805 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":529,"threads_started":1}
I20260620 02:40:22.952605 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:22.951313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:38221)
0620 02:40:22.952099 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:22.952379 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":318,"threads_started":1}
I20260620 02:40:23.265202 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:23.263781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:51721)
0620 02:40:23.264759 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:23.264874 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":362,"threads_started":1}
I20260620 02:40:23.566953 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:23.565738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:38449)
0620 02:40:23.566480 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:23.566596 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":347,"threads_started":1}
I20260620 02:40:23.955183 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:23.954145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:40021)
0620 02:40:23.954683 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:23.954988 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":300,"threads_started":1}
I20260620 02:40:24.267917 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:24.266752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:38311)
0620 02:40:24.267494 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:24.267609 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":304,"threads_started":1}
I20260620 02:40:24.570386 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:24.568568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:56785)
0620 02:40:24.569299 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:24.569424 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":444,"threads_started":1}
I20260620 02:40:24.959079 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:24.957061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:48847)
0620 02:40:24.958234 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:24.958394 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":970,"thread_start_us":660,"threads_started":1}
I20260620 02:40:25.270517 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:25.269542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:50639)
0620 02:40:25.270277 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:25.270353 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":458,"threads_started":1}
I20260620 02:40:25.573159 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:25.572243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:44663)
0620 02:40:25.572852 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:25.572961 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":348,"threads_started":1}
I20260620 02:40:25.961746 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:25.960618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:59729)
0620 02:40:25.961371 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:25.961503 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":373,"threads_started":1}
I20260620 02:40:26.272591 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:26.271648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:51599)
0620 02:40:26.272263 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:26.272361 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":341,"threads_started":1}
I20260620 02:40:26.575554 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:26.574405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:49849)
0620 02:40:26.575164 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:26.575364 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":546,"threads_started":1}
I20260620 02:40:26.964357 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:26.963077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:32935)
0620 02:40:26.963860 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:26.964105 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":325,"threads_started":1}
I20260620 02:40:27.275588 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:27.274323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:37487)
0620 02:40:27.275254 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:27.275384 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":444,"threads_started":1}
I20260620 02:40:27.578130 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:27.577141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:45267)
0620 02:40:27.577761 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:27.577879 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":354,"threads_started":1}
I20260620 02:40:27.966871 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:27.965843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:60575)
0620 02:40:27.966525 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:27.966635 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":366,"threads_started":1}
I20260620 02:40:28.279110 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:28.277618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:53973)
0620 02:40:28.278622 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:28.278780 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":667,"threads_started":1}
I20260620 02:40:28.580930 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:28.579913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:53225)
0620 02:40:28.580554 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:28.580648 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":332,"threads_started":1}
I20260620 02:40:28.969655 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:28.968594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:36075)
0620 02:40:28.969320 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:28.969429 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":448,"threads_started":1}
I20260620 02:40:29.282008 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:29.280621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:60355)
0620 02:40:29.281593 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:29.281745 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":584,"threads_started":1}
I20260620 02:40:29.583309 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:29.582167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:49501)
0620 02:40:29.582959 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:29.583076 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":356,"threads_started":1}
I20260620 02:40:29.972899 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:29.971692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:37791)
0620 02:40:29.972471 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:29.972575 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":582,"threads_started":1}
I20260620 02:40:30.284977 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:30.283705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:49341)
0620 02:40:30.284577 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:30.284683 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":615,"threads_started":1}
I20260620 02:40:30.586784 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:30.585354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:35789)
0620 02:40:30.586458 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:30.586566 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":692,"threads_started":1}
I20260620 02:40:30.976198 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:30.974034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:52649)
0620 02:40:30.975706 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:30.975807 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1563,"thread_start_us":1282,"threads_started":1}
I20260620 02:40:31.288487 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:31.287009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:50105)
0620 02:40:31.288039 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:31.288156 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":745,"threads_started":1}
I20260620 02:40:31.592263 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:31.589835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:50517)
0620 02:40:31.591382 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:31.591543 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":905,"spinlock_wait_cycles":615808,"thread_start_us":747,"threads_started":1}
I20260620 02:40:31.979437 17555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:31.977751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:33349)
0620 02:40:31.978729 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:31.978888 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":575,"threads_started":1}
I20260620 02:40:32.291857 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:32.290385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.2:50699)
0620 02:40:32.291320 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:32.291488 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":490,"threads_started":1}
I20260620 02:40:32.596284 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:32.593836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.1:42797)
0620 02:40:32.595165 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:32.595596 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":621,"threads_started":1}
I20260620 02:40:32.941501 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 17079
I20260620 02:40:32.971036 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:32.943075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.0.97.1:34435)
0620 02:40:32.944010 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:32.944047 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:32.944106 (+ 59us) server_negotiation.cc:407] Connection header received
0620 02:40:32.946471 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:32.946484 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:32.947032 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:32.947433 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:32.949814 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:32.951566 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:32.953671 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:32.955252 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:32.955547 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:32.960076 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:32.960092 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:32.960113 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:32.960179 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:32.964885 (+ 4706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:32.966300 (+ 1415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:32.966312 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:32.966324 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:32.966577 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:32.967648 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:32.967653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:32.967655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:32.968415 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:32.968716 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:32.970113 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0620 02:40:32.970650 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":362,"threads_started":1}
I20260620 02:40:32.972796 17242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:32.985260 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:32.982283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41621 (local address 127.0.97.3:60683)
0620 02:40:32.984620 (+ 2337us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:32.984787 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2132,"thread_start_us":1826,"threads_started":1}
I20260620 02:40:33.108619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 17079
I20260620 02:40:33.134599 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 17250
I20260620 02:40:33.161232 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:33.136355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.0.97.2:36877)
0620 02:40:33.137329 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:33.137343 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:33.137389 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:40:33.139135 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:33.139147 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:33.139662 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:33.140155 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:33.142007 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:33.143809 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:33.145838 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:33.147369 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:33.147685 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:33.152773 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:33.152785 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:33.152808 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:33.152887 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:33.157236 (+ 4349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:33.158158 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:33.158166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:33.158174 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:33.158376 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:33.158858 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:33.158863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:33.158866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:33.159495 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:33.159884 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:33.160432 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 02:40:33.160926 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":653,"threads_started":1}
I20260620 02:40:33.163527 17373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:33.272032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 17250
I20260620 02:40:33.293788 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 17381
I20260620 02:40:33.329123 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:33.295293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.0.97.3:44267)
0620 02:40:33.296487 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:33.296502 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:33.296554 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:40:33.298583 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:33.298598 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:33.299125 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:33.299567 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:33.302133 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:33.303582 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:33.305127 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:33.306820 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:33.307230 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:33.319110 (+ 11880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:33.319124 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:33.319145 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:33.319207 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:33.324578 (+ 5371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:33.325705 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:33.325719 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:33.325730 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:33.326024 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:33.326513 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:33.326518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:33.326520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:33.327163 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:33.327524 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:33.328087 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 02:40:33.328593 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":759,"threads_started":1}
I20260620 02:40:33.331297 17505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:33.439409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 17381
2026-06-20T02:40:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66764 ms)
[----------] 30 tests from SecurityITest (317664 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:40:33 dist-test-slave-rpg6 krb5kdc[17567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:40:33 dist-test-slave-rpg6 krb5kdc[17567](info): set up 2 sockets
Jun 20 02:40:33 dist-test-slave-rpg6 krb5kdc[17567](info): commencing operation
krb5kdc: starting...
W20260620 02:40:34.700393 388 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.006s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:40:34 dist-test-slave-rpg6 krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:40:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:40:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:40:35.032773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38157
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:40:35.461877 17580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:35.462241 17580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:35.462384 17580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:35.470686 17580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:40:35.470826 17580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:35.470918 17580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:35.471037 17580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:40:35.471112 17580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:40:35.485008 17580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:35.486876 17580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:35.488524 17580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:35.504117 17585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:35.504266 17586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:35.504359 17588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:35.505980 17580 server_base.cc:1034] running on GCE node
I20260620 02:40:35.507673 17580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:35.509961 17580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:35.511428 17580 hybrid_clock.cc:648] HybridClock initialized: now 1781923235511317 us; error 70 us; skew 500 ppm
I20260620 02:40:35.512208 17580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:35 dist-test-slave-rpg6 krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:35.517921 17580 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:40:35.520195 17580 webserver.cc:460] Webserver started at http://127.0.97.62:34999/ using document root <none> and password file <none>
I20260620 02:40:35.521052 17580 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:35.521238 17580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:35.521587 17580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:35.525110 17580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "aee0379ab1e8432896d0937aa811a34e"
format_stamp: "Formatted at 2026-06-20 02:40:35 on dist-test-slave-rpg6"
server_key: "d3ef41b46c5995b599bf5eb6fcdc5c63"
server_key_iv: "ceea04a697451d27260c862db3e2af80"
server_key_version: "encryptionkey@0"
I20260620 02:40:35.526050 17580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "aee0379ab1e8432896d0937aa811a34e"
format_stamp: "Formatted at 2026-06-20 02:40:35 on dist-test-slave-rpg6"
server_key: "d3ef41b46c5995b599bf5eb6fcdc5c63"
server_key_iv: "ceea04a697451d27260c862db3e2af80"
server_key_version: "encryptionkey@0"
I20260620 02:40:35.532580 17580 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 02:40:35.537699 17595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:35.539951 17580 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:40:35.540234 17580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "aee0379ab1e8432896d0937aa811a34e"
format_stamp: "Formatted at 2026-06-20 02:40:35 on dist-test-slave-rpg6"
server_key: "d3ef41b46c5995b599bf5eb6fcdc5c63"
server_key_iv: "ceea04a697451d27260c862db3e2af80"
server_key_version: "encryptionkey@0"
I20260620 02:40:35.540508 17580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:35.555984 17580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:35.561662 17580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:35.562206 17580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:35.589504 17580 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38157
I20260620 02:40:35.589534 17646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38157 every 8 connection(s)
I20260620 02:40:35.591284 17580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:40:35.596767 17647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:35.597606 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 17580
I20260620 02:40:35.597857 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:40:35.598821 388 external_mini_cluster.cc:1428] Setting key f9c56b9e4673bf9fb395749cd6f67649
I20260620 02:40:35.618410 17647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e: Bootstrap starting.
I20260620 02:40:35.623559 17647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e: Neither blocks nor log segments found. Creating new log.
Jun 20 02:40:35 dist-test-slave-rpg6 krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:40:35.625082 17647 log.cc:824] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:35.629869 17647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e: No bootstrap required, opened a new log
I20260620 02:40:35.635025 17650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:35.602228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.0.97.62:38157)
0620 02:40:35.603221 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:35.603243 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:35.603295 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:40:35.605913 (+ 2618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:35.605974 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:35.607003 (+ 1029us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:35.607483 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:35.610450 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:35.614672 (+ 4222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:35.617415 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:35.619460 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:35.619656 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:35.625931 (+ 6275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:35.625942 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:35.625964 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:35.626016 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:35.629630 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:35.630602 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:35.630610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:35.630618 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:35.630825 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:35.631345 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:35.631350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:35.631352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:35.632126 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:35.632491 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:35.633092 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 02:40:35.633683 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":456,"threads_started":1}
I20260620 02:40:35.640021 17647 raft_consensus.cc:348] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0379ab1e8432896d0937aa811a34e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38157 } }
I20260620 02:40:35.640564 17647 raft_consensus.cc:374] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:35.641017 17647 raft_consensus.cc:729] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee0379ab1e8432896d0937aa811a34e, State: Initialized, Role: FOLLOWER
I20260620 02:40:35.642154 17647 consensus_queue.cc:260] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0379ab1e8432896d0937aa811a34e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38157 } }
I20260620 02:40:35.642623 17647 raft_consensus.cc:388] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:40:35.642900 17647 raft_consensus.cc:482] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:40:35.643309 17647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:35.647239 17647 raft_consensus.cc:504] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0379ab1e8432896d0937aa811a34e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38157 } }
I20260620 02:40:35.648192 17647 leader_election.cc:302] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aee0379ab1e8432896d0937aa811a34e; no voters:
I20260620 02:40:35.648965 17647 leader_election.cc:288] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:40:35.649093 17652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:40:35.650484 17652 raft_consensus.cc:686] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [term 1 LEADER]: Becoming Leader. State: Replica: aee0379ab1e8432896d0937aa811a34e, State: Running, Role: LEADER
I20260620 02:40:35.651209 17652 consensus_queue.cc:237] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0379ab1e8432896d0937aa811a34e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38157 } }
I20260620 02:40:35.652665 17647 sys_catalog.cc:564] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:40:35.655223 17654 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [sys.catalog]: SysCatalogTable state changed. Reason: New leader aee0379ab1e8432896d0937aa811a34e. Latest consensus state: current_term: 1 leader_uuid: "aee0379ab1e8432896d0937aa811a34e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0379ab1e8432896d0937aa811a34e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38157 } } }
I20260620 02:40:35.655772 17654 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:35.656040 17653 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aee0379ab1e8432896d0937aa811a34e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee0379ab1e8432896d0937aa811a34e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38157 } } }
I20260620 02:40:35.657409 17653 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:35.662986 17659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:40:35.670169 17659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:40:35.681433 17659 catalog_manager.cc:1347] Generated new cluster ID: c9ff193832f84df88a013dfde599d9c7
I20260620 02:40:35.681612 17659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:40:35.696978 17659 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:40:35.698710 17659 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:40:35.707424 17659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e: Generated new TSK 0
I20260620 02:40:35.708680 17659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
W20260620 02:40:35.815554 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.1: real 0.100s user 0.001s sys 0.024s
I20260620 02:40:35.816817 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38157
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:40:36.327208 17673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:36.327546 17673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:36.327649 17673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:36.338527 17673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:36.338706 17673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:36.338842 17673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:40:36.353173 17673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:36.354846 17673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:36.356928 17673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:36.378769 17678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:36.379142 17681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:36.379209 17679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:36.381304 17673 server_base.cc:1034] running on GCE node
I20260620 02:40:36.382234 17673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:36.383442 17673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:36.387036 17673 hybrid_clock.cc:648] HybridClock initialized: now 1781923236386969 us; error 63 us; skew 500 ppm
I20260620 02:40:36.387523 17673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:36 dist-test-slave-rpg6 krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:36.392184 17673 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:40:36.394979 17673 webserver.cc:460] Webserver started at http://127.0.97.1:34899/ using document root <none> and password file <none>
I20260620 02:40:36.396284 17673 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:36.396494 17673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:36.396951 17673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:36.401811 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "27cce3fb3fbf444cbec7251871c15e01"
format_stamp: "Formatted at 2026-06-20 02:40:36 on dist-test-slave-rpg6"
server_key: "e0b6793cf1378ecd856a8c190a980bc7"
server_key_iv: "76274f8da93735de02314d5483d66002"
server_key_version: "encryptionkey@0"
I20260620 02:40:36.402895 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "27cce3fb3fbf444cbec7251871c15e01"
format_stamp: "Formatted at 2026-06-20 02:40:36 on dist-test-slave-rpg6"
server_key: "e0b6793cf1378ecd856a8c190a980bc7"
server_key_iv: "76274f8da93735de02314d5483d66002"
server_key_version: "encryptionkey@0"
I20260620 02:40:36.410272 17673 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 02:40:36.415009 17688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:36.417394 17673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 02:40:36.417652 17673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "27cce3fb3fbf444cbec7251871c15e01"
format_stamp: "Formatted at 2026-06-20 02:40:36 on dist-test-slave-rpg6"
server_key: "e0b6793cf1378ecd856a8c190a980bc7"
server_key_iv: "76274f8da93735de02314d5483d66002"
server_key_version: "encryptionkey@0"
I20260620 02:40:36.417925 17673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:36.448379 17673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:36.455554 17673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:36.456465 17673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:36.458293 17673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:36.460423 17673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:36.460565 17673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:36.461066 17673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:36.461184 17673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:36.515550 17673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42785
I20260620 02:40:36.515681 17800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42785 every 8 connection(s)
I20260620 02:40:36.517251 17673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:40:36.519340 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 17673
I20260620 02:40:36.519582 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:40:36.520351 388 external_mini_cluster.cc:1428] Setting key ca9c5316db1da4e7af40a63320b221ed
Jun 20 02:40:36 dist-test-slave-rpg6 krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:40:36.546793 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:36.521330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37931 (local address 127.0.97.62:38157)
0620 02:40:36.522035 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:36.522044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:36.524113 (+ 2069us) server_negotiation.cc:407] Connection header received
0620 02:40:36.527222 (+ 3109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:36.527229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:36.527353 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:36.527482 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:36.529399 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:36.530240 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:36.532319 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:36.532745 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:36.532905 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:36.537504 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:36.537510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:36.537514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:36.537559 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:36.542869 (+ 5310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:36.543972 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:36.543977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:36.543980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:36.544131 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:36.544693 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:36.544700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:36.544708 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:36.545033 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:36.545222 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:36.546063 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 02:40:36.546349 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":365,"threads_started":1}
I20260620 02:40:36.547964 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:36.522427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38157 (local address 127.0.97.1:37931)
0620 02:40:36.523875 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:36.523964 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:36.526580 (+ 2616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:36.527799 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:36.527810 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:36.528492 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:36.529169 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:36.529190 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:36.530462 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:36.530466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:36.532125 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:36.532144 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:36.533020 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:36.533028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:36.533306 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:36.534309 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:40:36.534351 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:36.537152 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:36.543147 (+ 5995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:36.543156 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:36.543174 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:36.543727 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:36.544268 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:36.544273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:36.544279 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:36.544546 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:36.545195 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:36.545204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:36.545628 (+ 424us) client_negotiation.cc:769] Sending connection context
0620 02:40:36.545966 (+ 338us) client_negotiation.cc:240] Negotiation successful
0620 02:40:36.546617 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"thread_start_us":517,"threads_started":1}
I20260620 02:40:36.550465 17801 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38157
I20260620 02:40:36.550959 17801 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:36.551957 17801 heartbeater.cc:511] Master 127.0.97.62:38157 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:40:36.557600 17612 ts_manager.cc:194] Registered new tserver with Master: 27cce3fb3fbf444cbec7251871c15e01 (127.0.97.1:42785)
I20260620 02:40:36.560731 17612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:40:36.608680 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38157
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:40:37.064499 17807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:37.065148 17807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:37.065279 17807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:37.076282 17807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:37.076442 17807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:37.076655 17807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:40:37.090306 17807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:37.092341 17807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:37.093895 17807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:37.116832 17812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:37.117250 17815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:37.117278 17813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:37.118942 17807 server_base.cc:1034] running on GCE node
I20260620 02:40:37.120631 17807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:37.122515 17807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:37.123934 17807 hybrid_clock.cc:648] HybridClock initialized: now 1781923237123766 us; error 80 us; skew 500 ppm
I20260620 02:40:37.124490 17807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:37 dist-test-slave-rpg6 krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:37.129384 17807 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:40:37.131529 17807 webserver.cc:460] Webserver started at http://127.0.97.2:38331/ using document root <none> and password file <none>
I20260620 02:40:37.132483 17807 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:37.132746 17807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:37.133316 17807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:37.137306 17807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "558315964782453ea33de61861a6536f"
format_stamp: "Formatted at 2026-06-20 02:40:37 on dist-test-slave-rpg6"
server_key: "dc570cd5c4350c7f4b24fbec74fec086"
server_key_iv: "934f3529f89d0f6e693981559e9fc758"
server_key_version: "encryptionkey@0"
I20260620 02:40:37.138695 17807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "558315964782453ea33de61861a6536f"
format_stamp: "Formatted at 2026-06-20 02:40:37 on dist-test-slave-rpg6"
server_key: "dc570cd5c4350c7f4b24fbec74fec086"
server_key_iv: "934f3529f89d0f6e693981559e9fc758"
server_key_version: "encryptionkey@0"
I20260620 02:40:37.149091 17807 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20260620 02:40:37.155385 17822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:37.159188 17807 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260620 02:40:37.160058 17807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "558315964782453ea33de61861a6536f"
format_stamp: "Formatted at 2026-06-20 02:40:37 on dist-test-slave-rpg6"
server_key: "dc570cd5c4350c7f4b24fbec74fec086"
server_key_iv: "934f3529f89d0f6e693981559e9fc758"
server_key_version: "encryptionkey@0"
I20260620 02:40:37.160418 17807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:37.180936 17807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:37.188335 17807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:37.188879 17807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:37.190603 17807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:37.192655 17807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:37.192795 17807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:37.192965 17807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:37.193040 17807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:37.248839 17807 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46349
I20260620 02:40:37.248922 17934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46349 every 8 connection(s)
I20260620 02:40:37.250823 17807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:40:37.253026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 17807
I20260620 02:40:37.253248 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:40:37.253829 388 external_mini_cluster.cc:1428] Setting key f67d26ffee1f2655610ed1c65ed4eaac
Jun 20 02:40:37 dist-test-slave-rpg6 krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:40:37.286923 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:37.255589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59797 (local address 127.0.97.62:38157)
0620 02:40:37.256279 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:37.256286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:37.257782 (+ 1496us) server_negotiation.cc:407] Connection header received
0620 02:40:37.261915 (+ 4133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:37.261920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:37.262196 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:37.262331 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:37.265345 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:37.266333 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:37.269339 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:37.270347 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:37.270537 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:37.275103 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:37.275114 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:37.275122 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:37.275184 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:37.280340 (+ 5156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:37.281544 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:37.281554 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:37.281561 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:37.281780 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:37.282905 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:37.282915 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:37.282921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:37.283673 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:37.283947 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:37.285568 (+ 1621us) server_negotiation.cc:299] Negotiation successful
0620 02:40:37.286248 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":432,"threads_started":1}
I20260620 02:40:37.287101 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:37.256377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38157 (local address 127.0.97.2:59797)
0620 02:40:37.257486 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:37.257555 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:37.261332 (+ 3777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:37.262772 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:37.262825 (+ 53us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:37.263926 (+ 1101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:37.264951 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:37.264991 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:37.266573 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:37.266578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:37.269094 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:37.269120 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:37.270650 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:37.270657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:37.271035 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:37.271968 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:40:37.271996 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:37.274713 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:37.280676 (+ 5963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:37.280686 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:37.280704 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:37.281286 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:37.282155 (+ 869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:37.282162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:37.282165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:37.282663 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:37.284168 (+ 1505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:37.284182 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:37.284697 (+ 515us) client_negotiation.cc:769] Sending connection context
0620 02:40:37.285304 (+ 607us) client_negotiation.cc:240] Negotiation successful
0620 02:40:37.285812 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":477,"threads_started":1}
I20260620 02:40:37.291435 17935 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38157
I20260620 02:40:37.292074 17935 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:40:37.293221 17935 heartbeater.cc:511] Master 127.0.97.62:38157 requested a full tablet report, sending...
I20260620 02:40:37.296007 17612 ts_manager.cc:194] Registered new tserver with Master: 558315964782453ea33de61861a6536f (127.0.97.2:46349)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:40:37.297351 17612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:40:37.350286 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38157
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:40:37.565095 17801 heartbeater.cc:503] Master 127.0.97.62:38157 was elected leader, sending a full tablet report...
W20260620 02:40:37.759024 17941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:37.759743 17941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:37.760030 17941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:37.772250 17941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:37.772485 17941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:37.772704 17941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:40:37.787360 17941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:37.789268 17941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:37.791147 17941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:37.808456 17946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:37.808451 17949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:37.808540 17947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:37.809544 17941 server_base.cc:1034] running on GCE node
I20260620 02:40:37.810742 17941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:37.812255 17941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:37.813588 17941 hybrid_clock.cc:648] HybridClock initialized: now 1781923237813540 us; error 50 us; skew 500 ppm
I20260620 02:40:37.814126 17941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:37 dist-test-slave-rpg6 krb5kdc[17567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:37.818799 17941 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:40:37.821954 17941 webserver.cc:460] Webserver started at http://127.0.97.3:40767/ using document root <none> and password file <none>
I20260620 02:40:37.823045 17941 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:37.823266 17941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:37.824306 17941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:37.828545 17941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "f2a0efddfa9f4d718a7ef0b276bb9107"
format_stamp: "Formatted at 2026-06-20 02:40:37 on dist-test-slave-rpg6"
server_key: "171f56725c493958fa4d9a4f4d805fa7"
server_key_iv: "2c4f501d334ad8df23632fc6050d9229"
server_key_version: "encryptionkey@0"
I20260620 02:40:37.829595 17941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f2a0efddfa9f4d718a7ef0b276bb9107"
format_stamp: "Formatted at 2026-06-20 02:40:37 on dist-test-slave-rpg6"
server_key: "171f56725c493958fa4d9a4f4d805fa7"
server_key_iv: "2c4f501d334ad8df23632fc6050d9229"
server_key_version: "encryptionkey@0"
I20260620 02:40:37.837419 17941 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260620 02:40:37.842422 17956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:37.845355 17941 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:40:37.845888 17941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "f2a0efddfa9f4d718a7ef0b276bb9107"
format_stamp: "Formatted at 2026-06-20 02:40:37 on dist-test-slave-rpg6"
server_key: "171f56725c493958fa4d9a4f4d805fa7"
server_key_iv: "2c4f501d334ad8df23632fc6050d9229"
server_key_version: "encryptionkey@0"
I20260620 02:40:37.846138 17941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:37.868990 17941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:37.876515 17941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:37.876901 17941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:37.878202 17941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:37.879956 17941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:37.880055 17941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:37.880188 17941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:37.880270 17941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:37.940961 17941 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34381
I20260620 02:40:37.941094 18068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34381 every 8 connection(s)
I20260620 02:40:37.942994 17941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:40:37.946041 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 17941
I20260620 02:40:37.946266 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:40:37.946832 388 external_mini_cluster.cc:1428] Setting key 3d357c5876631372d067b06567aa758d
Jun 20 02:40:37 dist-test-slave-rpg6 krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:40:37.976318 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:37.946957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47835 (local address 127.0.97.62:38157)
0620 02:40:37.947876 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:37.947883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:37.949440 (+ 1557us) server_negotiation.cc:407] Connection header received
0620 02:40:37.953236 (+ 3796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:37.953241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:37.953383 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:37.953510 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:37.955259 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:37.956163 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:37.957898 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:37.958367 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:37.958519 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:37.965323 (+ 6804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:37.965410 (+ 87us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:37.965422 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:37.965536 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:37.970650 (+ 5114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:37.972171 (+ 1521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:37.972178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:37.972180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:37.972360 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:37.972995 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:37.973001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:37.973004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:37.973957 (+ 953us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:37.974111 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:37.975453 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0620 02:40:37.975746 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":301,"threads_started":1}
I20260620 02:40:37.977741 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:37.947914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38157 (local address 127.0.97.3:47835)
0620 02:40:37.949262 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:37.949317 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:37.952860 (+ 3543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:37.953801 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:37.953812 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:37.954397 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:37.955034 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:37.955053 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:37.956354 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:37.956358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:37.957714 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:37.957728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:37.958631 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:37.958641 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:37.958909 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:37.960227 (+ 1318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:40:37.960274 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:37.964407 (+ 4133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:37.970919 (+ 6512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:37.970931 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:37.970955 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:37.971627 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:37.972526 (+ 899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:37.972530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:37.972533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:37.972753 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:37.974245 (+ 1492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:37.974256 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:37.974836 (+ 580us) client_negotiation.cc:769] Sending connection context
0620 02:40:37.975262 (+ 426us) client_negotiation.cc:240] Negotiation successful
0620 02:40:37.976137 (+ 875us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":598,"threads_started":1}
I20260620 02:40:37.980180 18069 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38157
I20260620 02:40:37.980719 18069 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:37.981801 18069 heartbeater.cc:511] Master 127.0.97.62:38157 requested a full tablet report, sending...
I20260620 02:40:37.984154 17612 ts_manager.cc:194] Registered new tserver with Master: f2a0efddfa9f4d718a7ef0b276bb9107 (127.0.97.3:34381)
I20260620 02:40:37.985092 17612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47835
I20260620 02:40:37.994453 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:40:38.018136 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.000163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.0.97.62:38157)
0620 02:40:38.000477 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:38.000481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:38.000873 (+ 392us) server_negotiation.cc:407] Connection header received
0620 02:40:38.001000 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:38.001005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:38.001079 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:38.001202 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:38.003139 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.004433 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.006268 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.007014 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.007234 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.010495 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:38.010503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:38.010506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:38.010557 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:38.013823 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:38.014818 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:38.014823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:38.014826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:38.015014 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:38.015693 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:38.015697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:38.015700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:38.016235 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:38.016467 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:38.017399 (+ 932us) server_negotiation.cc:299] Negotiation successful
0620 02:40:38.017660 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260620 02:40:38.027668 17612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:40:38.036947 17612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:40:38.078469 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.056170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.0.97.2:46349)
0620 02:40:38.057209 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:38.057219 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:38.059278 (+ 2059us) server_negotiation.cc:407] Connection header received
0620 02:40:38.059897 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:38.059907 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:38.060250 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:38.060472 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:38.061768 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.064454 (+ 2686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.071240 (+ 6786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.073850 (+ 2610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.074036 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.074324 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:38.077544 (+ 3220us) server_negotiation.cc:299] Negotiation successful
0620 02:40:38.077952 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":490,"threads_started":1}
I20260620 02:40:38.080114 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.055909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46349 (local address 127.0.0.1:58766)
0620 02:40:38.058963 (+ 3054us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:38.059029 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:38.059487 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:38.060875 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:38.060883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:38.060914 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:38.061509 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:38.061530 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.064844 (+ 3314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.064853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:38.070792 (+ 5939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:38.070825 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.076192 (+ 5367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.076210 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:38.076466 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.076523 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 02:40:38.079141 (+ 2618us) client_negotiation.cc:240] Negotiation successful
0620 02:40:38.079457 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"mutex_wait_us":1,"thread_start_us":1919,"threads_started":1}
I20260620 02:40:38.081436 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.058310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34381 (local address 127.0.0.1:44298)
0620 02:40:38.060464 (+ 2154us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:38.060485 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:38.060634 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:38.062037 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:38.062046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:38.062084 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:38.062721 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:38.062739 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.068098 (+ 5359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.068109 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:38.075582 (+ 7473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:38.075620 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.080277 (+ 4657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.080285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:38.080518 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.080551 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:40:38.080816 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 02:40:38.081003 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1994,"thread_start_us":1703,"threads_started":1}
I20260620 02:40:38.083101 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.059136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.0.97.3:34381)
0620 02:40:38.061290 (+ 2154us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:38.061302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:38.061343 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:40:38.061449 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:38.061459 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:38.061760 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:38.063033 (+ 1273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:38.063421 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.067576 (+ 4155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.076197 (+ 8621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.079973 (+ 3776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.081754 (+ 1781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.082051 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:38.082172 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 02:40:38.082614 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1928,"thread_start_us":2936,"threads_started":1}
I20260620 02:40:38.087261 17870 tablet_service.cc:1467] Processing CreateTablet for tablet 39a9b2bad0694dd1a9382aa6e71ecf93 (DEFAULT_TABLE table=test-table [id=2141a89d785a4cb8beb297513a8dcf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:38.090113 18004 tablet_service.cc:1467] Processing CreateTablet for tablet 39a9b2bad0694dd1a9382aa6e71ecf93 (DEFAULT_TABLE table=test-table [id=2141a89d785a4cb8beb297513a8dcf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:38.092702 17870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a9b2bad0694dd1a9382aa6e71ecf93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:38.093468 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a9b2bad0694dd1a9382aa6e71ecf93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:38.097852 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.061580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42785 (local address 127.0.0.1:58392)
0620 02:40:38.076344 (+ 14764us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:38.076370 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:38.078152 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:38.080009 (+ 1857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:38.080017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:38.080058 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:38.080494 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:38.080507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.086234 (+ 5727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.086242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:38.090058 (+ 3816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:38.091533 (+ 1475us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.096089 (+ 4556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.096098 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:38.096945 (+ 847us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.096989 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 02:40:38.097104 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 02:40:38.097403 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14577,"thread_start_us":13819,"threads_started":1}
I20260620 02:40:38.098352 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.061851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.0.97.1:42785)
0620 02:40:38.072732 (+ 10881us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:38.072743 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:38.078384 (+ 5641us) server_negotiation.cc:407] Connection header received
0620 02:40:38.079131 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:38.079142 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:38.079687 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:38.080017 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:38.083499 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.085838 (+ 2339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.092012 (+ 6174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.095042 (+ 3030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.095250 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.097640 (+ 2390us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:38.097754 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 02:40:38.097970 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10636,"thread_start_us":10912,"threads_started":1}
I20260620 02:40:38.103649 17736 tablet_service.cc:1467] Processing CreateTablet for tablet 39a9b2bad0694dd1a9382aa6e71ecf93 (DEFAULT_TABLE table=test-table [id=2141a89d785a4cb8beb297513a8dcf09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:38.107784 17736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39a9b2bad0694dd1a9382aa6e71ecf93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:38.118229 18088 tablet_bootstrap.cc:492] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: Bootstrap starting.
I20260620 02:40:38.120618 18089 tablet_bootstrap.cc:492] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: Bootstrap starting.
I20260620 02:40:38.125156 18089 tablet_bootstrap.cc:654] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:38.125582 18088 tablet_bootstrap.cc:654] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:38.126807 18089 log.cc:824] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:38.126998 18088 log.cc:824] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:38.131883 18092 tablet_bootstrap.cc:492] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: Bootstrap starting.
I20260620 02:40:38.132263 18088 tablet_bootstrap.cc:492] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: No bootstrap required, opened a new log
I20260620 02:40:38.132279 18089 tablet_bootstrap.cc:492] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: No bootstrap required, opened a new log
I20260620 02:40:38.132587 18089 ts_tablet_manager.cc:1397] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.004s
I20260620 02:40:38.132691 18088 ts_tablet_manager.cc:1397] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260620 02:40:38.137190 18092 tablet_bootstrap.cc:654] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:38.138875 18092 log.cc:824] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:38.143769 18092 tablet_bootstrap.cc:492] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: No bootstrap required, opened a new log
I20260620 02:40:38.144157 18092 ts_tablet_manager.cc:1397] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260620 02:40:38.143702 18089 raft_consensus.cc:348] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558315964782453ea33de61861a6536f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46349 } } peers { permanent_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34381 } } peers { permanent_uuid: "27cce3fb3fbf444cbec7251871c15e01" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42785 } }
I20260620 02:40:38.144383 18089 raft_consensus.cc:374] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:38.144500 18089 raft_consensus.cc:729] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558315964782453ea33de61861a6536f, State: Initialized, Role: FOLLOWER
I20260620 02:40:38.144567 18088 raft_consensus.cc:348] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558315964782453ea33de61861a6536f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46349 } } peers { permanent_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34381 } } peers { permanent_uuid: "27cce3fb3fbf444cbec7251871c15e01" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42785 } }
I20260620 02:40:38.145431 18088 raft_consensus.cc:374] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:38.145599 18088 raft_consensus.cc:729] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a0efddfa9f4d718a7ef0b276bb9107, State: Initialized, Role: FOLLOWER
I20260620 02:40:38.145790 18089 consensus_queue.cc:260] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558315964782453ea33de61861a6536f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46349 } } peers { permanent_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34381 } } peers { permanent_uuid: "27cce3fb3fbf444cbec7251871c15e01" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42785 } }
I20260620 02:40:38.146301 18088 consensus_queue.cc:260] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558315964782453ea33de61861a6536f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46349 } } peers { permanent_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34381 } } peers { permanent_uuid: "27cce3fb3fbf444cbec7251871c15e01" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42785 } }
I20260620 02:40:38.147233 17935 heartbeater.cc:503] Master 127.0.97.62:38157 was elected leader, sending a full tablet report...
I20260620 02:40:38.148138 18089 ts_tablet_manager.cc:1428] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260620 02:40:38.149297 18069 heartbeater.cc:503] Master 127.0.97.62:38157 was elected leader, sending a full tablet report...
I20260620 02:40:38.150286 18088 ts_tablet_manager.cc:1428] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: Time spent starting tablet: real 0.017s user 0.009s sys 0.006s
I20260620 02:40:38.158859 18092 raft_consensus.cc:348] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558315964782453ea33de61861a6536f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46349 } } peers { permanent_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34381 } } peers { permanent_uuid: "27cce3fb3fbf444cbec7251871c15e01" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42785 } }
I20260620 02:40:38.159366 18092 raft_consensus.cc:374] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:38.159811 18092 raft_consensus.cc:729] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27cce3fb3fbf444cbec7251871c15e01, State: Initialized, Role: FOLLOWER
I20260620 02:40:38.161049 18092 consensus_queue.cc:260] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558315964782453ea33de61861a6536f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46349 } } peers { permanent_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34381 } } peers { permanent_uuid: "27cce3fb3fbf444cbec7251871c15e01" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42785 } }
I20260620 02:40:38.163208 18092 ts_tablet_manager.cc:1428] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
W20260620 02:40:38.195379 18070 tablet.cc:2369] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:40:38.254597 17936 tablet.cc:2369] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:40:38.271184 17802 tablet.cc:2369] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:40:38.422845 18096 raft_consensus.cc:482] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:40:38.423130 18096 raft_consensus.cc:504] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558315964782453ea33de61861a6536f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46349 } } peers { permanent_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34381 } } peers { permanent_uuid: "27cce3fb3fbf444cbec7251871c15e01" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42785 } }
I20260620 02:40:38.425374 18096 leader_election.cc:288] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 558315964782453ea33de61861a6536f (127.0.97.2:46349), f2a0efddfa9f4d718a7ef0b276bb9107 (127.0.97.3:34381)
I20260620 02:40:38.432865 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.425824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51605 (local address 127.0.97.2:46349)
0620 02:40:38.426238 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:38.426245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:38.426518 (+ 273us) server_negotiation.cc:407] Connection header received
0620 02:40:38.426877 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:38.426881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:38.426944 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:38.427054 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:38.428051 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.428787 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.430859 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.431708 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.431980 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.432093 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:38.432422 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 02:40:38.432562 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260620 02:40:38.433002 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.425597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46349 (local address 127.0.97.1:51605)
0620 02:40:38.426372 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:38.426417 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:38.426651 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:38.427068 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:38.427072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:38.427108 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:38.427787 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:38.427802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.428950 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.428959 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:38.430621 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:38.430640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.432054 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.432060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:38.432236 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.432262 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 02:40:38.432394 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 02:40:38.432547 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":2,"thread_start_us":822,"threads_started":1}
I20260620 02:40:38.433127 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.425970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46635 (local address 127.0.97.3:34381)
0620 02:40:38.426391 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:38.426417 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:38.426769 (+ 352us) server_negotiation.cc:407] Connection header received
0620 02:40:38.426957 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:38.426962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:38.427022 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:38.427127 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:38.428375 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.429200 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.431199 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.432125 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.432289 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.432403 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:38.432722 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 02:40:38.432868 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298}
I20260620 02:40:38.433472 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.425735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34381 (local address 127.0.97.1:46635)
0620 02:40:38.426661 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:38.426676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:38.426785 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:38.427143 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:38.427147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:38.427177 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:38.428041 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:38.428060 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.429441 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.429451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:38.430999 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:38.431013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.432377 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.432386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:38.432551 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.432575 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 02:40:38.432687 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:40:38.432819 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":29,"thread_start_us":550,"threads_started":1}
I20260620 02:40:38.434314 17890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a9b2bad0694dd1a9382aa6e71ecf93" candidate_uuid: "27cce3fb3fbf444cbec7251871c15e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "558315964782453ea33de61861a6536f" is_pre_election: true
I20260620 02:40:38.434450 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a9b2bad0694dd1a9382aa6e71ecf93" candidate_uuid: "27cce3fb3fbf444cbec7251871c15e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" is_pre_election: true
I20260620 02:40:38.434933 18024 raft_consensus.cc:2454] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27cce3fb3fbf444cbec7251871c15e01 in term 0.
I20260620 02:40:38.434924 17890 raft_consensus.cc:2454] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27cce3fb3fbf444cbec7251871c15e01 in term 0.
I20260620 02:40:38.435874 17689 leader_election.cc:302] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27cce3fb3fbf444cbec7251871c15e01, f2a0efddfa9f4d718a7ef0b276bb9107; no voters:
I20260620 02:40:38.436685 18096 raft_consensus.cc:2781] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:40:38.436992 18096 raft_consensus.cc:482] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:40:38.437382 18096 raft_consensus.cc:3037] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:38.440045 18096 raft_consensus.cc:504] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558315964782453ea33de61861a6536f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46349 } } peers { permanent_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34381 } } peers { permanent_uuid: "27cce3fb3fbf444cbec7251871c15e01" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42785 } }
I20260620 02:40:38.440761 18096 leader_election.cc:288] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [CANDIDATE]: Term 1 election: Requested vote from peers 558315964782453ea33de61861a6536f (127.0.97.2:46349), f2a0efddfa9f4d718a7ef0b276bb9107 (127.0.97.3:34381)
I20260620 02:40:38.442298 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a9b2bad0694dd1a9382aa6e71ecf93" candidate_uuid: "27cce3fb3fbf444cbec7251871c15e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107"
I20260620 02:40:38.442296 17890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a9b2bad0694dd1a9382aa6e71ecf93" candidate_uuid: "27cce3fb3fbf444cbec7251871c15e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "558315964782453ea33de61861a6536f"
I20260620 02:40:38.442790 17890 raft_consensus.cc:3037] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:38.442791 18024 raft_consensus.cc:3037] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:38.447774 17890 raft_consensus.cc:2454] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27cce3fb3fbf444cbec7251871c15e01 in term 1.
I20260620 02:40:38.447774 18024 raft_consensus.cc:2454] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27cce3fb3fbf444cbec7251871c15e01 in term 1.
I20260620 02:40:38.448879 17692 leader_election.cc:302] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27cce3fb3fbf444cbec7251871c15e01, 558315964782453ea33de61861a6536f; no voters:
I20260620 02:40:38.450197 18096 raft_consensus.cc:2781] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:40:38.450860 18096 raft_consensus.cc:686] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 1 LEADER]: Becoming Leader. State: Replica: 27cce3fb3fbf444cbec7251871c15e01, State: Running, Role: LEADER
I20260620 02:40:38.451901 18096 consensus_queue.cc:237] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558315964782453ea33de61861a6536f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46349 } } peers { permanent_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34381 } } peers { permanent_uuid: "27cce3fb3fbf444cbec7251871c15e01" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42785 } }
I20260620 02:40:38.457959 17612 catalog_manager.cc:5495] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27cce3fb3fbf444cbec7251871c15e01 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "27cce3fb3fbf444cbec7251871c15e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558315964782453ea33de61861a6536f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27cce3fb3fbf444cbec7251871c15e01" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42785 } health_report { overall_health: HEALTHY } } }
I20260620 02:40:38.555166 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.546526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.0.97.1:42785)
0620 02:40:38.546848 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:38.546853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:38.547302 (+ 449us) server_negotiation.cc:407] Connection header received
0620 02:40:38.547595 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:38.547600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:38.547727 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:38.548003 (+ 276us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:40:38.548702 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.549655 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.552034 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.552658 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.552793 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.553626 (+ 833us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:40:38.554389 (+ 763us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:40:38.554561 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:38.554776 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 02:40:38.554887 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 02:40:38.569851 18024 raft_consensus.cc:1261] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107 [term 1 FOLLOWER]: Refusing update from remote peer 27cce3fb3fbf444cbec7251871c15e01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:40:38.570291 17890 raft_consensus.cc:1261] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f [term 1 FOLLOWER]: Refusing update from remote peer 27cce3fb3fbf444cbec7251871c15e01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:40:38.571255 18099 consensus_queue.cc:1035] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a0efddfa9f4d718a7ef0b276bb9107" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:40:38.571707 18096 consensus_queue.cc:1035] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558315964782453ea33de61861a6536f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:40:38.618057 17611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:40:38.618387 17611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:40:38.624464 17611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aee0379ab1e8432896d0937aa811a34e: Sending DeleteTablet for 3 replicas of tablet 39a9b2bad0694dd1a9382aa6e71ecf93
I20260620 02:40:38.626163 18004 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a9b2bad0694dd1a9382aa6e71ecf93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44298
I20260620 02:40:38.626169 17736 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a9b2bad0694dd1a9382aa6e71ecf93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58392
I20260620 02:40:38.626160 17870 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a9b2bad0694dd1a9382aa6e71ecf93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58766
I20260620 02:40:38.633579 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 17673
I20260620 02:40:38.637444 18115 tablet_replica.cc:330] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: stopping tablet replica
I20260620 02:40:38.638130 18115 raft_consensus.cc:2229] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:40:38.638621 18114 tablet_replica.cc:330] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: stopping tablet replica
I20260620 02:40:38.638748 18115 raft_consensus.cc:2258] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:38.639532 18114 raft_consensus.cc:2229] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:40:38.640967 18114 raft_consensus.cc:2258] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:38.642629 18113 tablet_replica.cc:330] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: stopping tablet replica
I20260620 02:40:38.644949 18114 ts_tablet_manager.cc:1905] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:38.645493 18113 raft_consensus.cc:2229] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:40:38.646772 18115 ts_tablet_manager.cc:1905] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:38.647759 18113 raft_consensus.cc:2258] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:38.655097 18114 ts_tablet_manager.cc:1918] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:38.655654 18114 log.cc:1196] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/39a9b2bad0694dd1a9382aa6e71ecf93
I20260620 02:40:38.656514 18114 ts_tablet_manager.cc:1939] T 39a9b2bad0694dd1a9382aa6e71ecf93 P f2a0efddfa9f4d718a7ef0b276bb9107: Deleting consensus metadata
I20260620 02:40:38.656512 18115 ts_tablet_manager.cc:1918] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:38.656695 18115 log.cc:1196] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/39a9b2bad0694dd1a9382aa6e71ecf93
I20260620 02:40:38.659392 17596 catalog_manager.cc:4841] TS f2a0efddfa9f4d718a7ef0b276bb9107 (127.0.97.3:34381): tablet 39a9b2bad0694dd1a9382aa6e71ecf93 (table test-table [id=2141a89d785a4cb8beb297513a8dcf09]) successfully deleted
I20260620 02:40:38.659480 18113 ts_tablet_manager.cc:1905] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:38.660382 18115 ts_tablet_manager.cc:1939] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 558315964782453ea33de61861a6536f: Deleting consensus metadata
I20260620 02:40:38.662518 17599 catalog_manager.cc:4841] TS 558315964782453ea33de61861a6536f (127.0.97.2:46349): tablet 39a9b2bad0694dd1a9382aa6e71ecf93 (table test-table [id=2141a89d785a4cb8beb297513a8dcf09]) successfully deleted
I20260620 02:40:38.666222 18113 ts_tablet_manager.cc:1918] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 20 02:40:38 dist-test-slave-rpg6 krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:40:38.666553 18113 log.cc:1196] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/39a9b2bad0694dd1a9382aa6e71ecf93
I20260620 02:40:38.667176 18113 ts_tablet_manager.cc:1939] T 39a9b2bad0694dd1a9382aa6e71ecf93 P 27cce3fb3fbf444cbec7251871c15e01: Deleting consensus metadata
I20260620 02:40:38.669183 17598 catalog_manager.cc:4841] TS 27cce3fb3fbf444cbec7251871c15e01 (127.0.97.1:42785): tablet 39a9b2bad0694dd1a9382aa6e71ecf93 (table test-table [id=2141a89d785a4cb8beb297513a8dcf09]) successfully deleted
I20260620 02:40:38.677711 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.637613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58412 (local address 127.0.97.1:42785)
0620 02:40:38.638562 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:38.638569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:38.649550 (+ 10981us) server_negotiation.cc:407] Connection header received
0620 02:40:38.650073 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:38.650080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:38.650172 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:38.650251 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:38.652545 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.653843 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.656892 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.657766 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.657891 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.668121 (+ 10230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:38.668140 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:38.668172 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:38.668232 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:38.672771 (+ 4539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:38.673927 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:38.673938 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:38.673942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:38.674493 (+ 551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:38.675229 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:38.675237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:38.675241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:38.676147 (+ 906us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:38.676600 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:38.676942 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 02:40:38.677271 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787}
I20260620 02:40:38.678561 17796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:38.800460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 17673
I20260620 02:40:38.826516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 17807
Jun 20 02:40:38 dist-test-slave-rpg6 krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:40:38.849913 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.827714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.0.97.2:46349)
0620 02:40:38.828034 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:38.828039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:38.828189 (+ 150us) server_negotiation.cc:407] Connection header received
0620 02:40:38.828353 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:38.828359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:38.828446 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:38.828568 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:38.830252 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.831324 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.833465 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.834142 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.834260 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:38.841161 (+ 6901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:38.841179 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:38.841196 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:38.841250 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:38.846134 (+ 4884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:38.847305 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:38.847316 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:38.847319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:38.847511 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:38.848100 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:38.848104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:38.848106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:38.848550 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:38.848788 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:38.849242 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 02:40:38.849525 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260620 02:40:38.850880 17930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:38.968863 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 17807
I20260620 02:40:38.992012 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 17941
Jun 20 02:40:39 dist-test-slave-rpg6 krb5kdc[17567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:40:39.016103 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:38.993531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.0.97.3:34381)
0620 02:40:38.994258 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:38.994263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:38.994285 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:40:38.994347 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:38.994351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:38.994415 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:38.994632 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:38.996767 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:38.997748 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:38.999720 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:39.000550 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:39.000719 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:39.007556 (+ 6837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:39.007575 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:39.007599 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:39.007654 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:39.011332 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:39.012561 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:39.012571 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:39.012574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:39.012752 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:39.013505 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:39.013510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:39.013512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:39.014286 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:39.014576 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:39.015144 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 02:40:39.015493 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":426,"threads_started":1}
I20260620 02:40:39.017215 18064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:39.150379 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 17941
I20260620 02:40:39.170552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 17580
I20260620 02:40:39.172369 17642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:39.312794 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 17580
2026-06-20T02:40:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5874 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T02:40:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:40:39Z Disabled control of system clock
I20260620 02:40:39.409544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36749
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:40:39.876741 18125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:39.877111 18125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:39.877252 18125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:39.887061 18125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:40:39.887287 18125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:39.887341 18125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:39.887516 18125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:40:39.887794 18125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:40:39.903730 18125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:39.905328 18125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:39.908134 18125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 02:40:39.927229 18125 server_base.cc:1034] running on GCE node
W20260620 02:40:39.927150 18133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:39.928228 18130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:39.927211 18131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:39.930388 18125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:39.932346 18125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:39.933848 18125 hybrid_clock.cc:648] HybridClock initialized: now 1781923239933747 us; error 79 us; skew 500 ppm
I20260620 02:40:39.934504 18125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:40:39.937642 18125 webserver.cc:460] Webserver started at http://127.0.97.62:42155/ using document root <none> and password file <none>
I20260620 02:40:39.938534 18125 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:39.938871 18125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:39.939253 18125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:39.942842 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "3fed2ffebc7b4977a87be7dc8712ca1b"
format_stamp: "Formatted at 2026-06-20 02:40:39 on dist-test-slave-rpg6"
server_key: "e595c7ed0613710d2edc2cb7e58c0624"
server_key_iv: "20792d966256233ad8c23546cde37cf4"
server_key_version: "encryptionkey@0"
I20260620 02:40:39.944761 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "3fed2ffebc7b4977a87be7dc8712ca1b"
format_stamp: "Formatted at 2026-06-20 02:40:39 on dist-test-slave-rpg6"
server_key: "e595c7ed0613710d2edc2cb7e58c0624"
server_key_iv: "20792d966256233ad8c23546cde37cf4"
server_key_version: "encryptionkey@0"
I20260620 02:40:39.954802 18125 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260620 02:40:39.960265 18139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:39.962117 18125 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:40:39.962575 18125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "3fed2ffebc7b4977a87be7dc8712ca1b"
format_stamp: "Formatted at 2026-06-20 02:40:39 on dist-test-slave-rpg6"
server_key: "e595c7ed0613710d2edc2cb7e58c0624"
server_key_iv: "20792d966256233ad8c23546cde37cf4"
server_key_version: "encryptionkey@0"
I20260620 02:40:39.963047 18125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:39.993290 18125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:39.994366 18125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:39.994696 18125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:40.020859 18125 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36749
I20260620 02:40:40.021124 18190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36749 every 8 connection(s)
I20260620 02:40:40.022468 18125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:40:40.026862 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 18125
I20260620 02:40:40.027071 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:40:40.027632 388 external_mini_cluster.cc:1428] Setting key cfbfedc72c395b2704f6069dcfa62c0e
I20260620 02:40:40.027833 18191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:40.045977 18191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b: Bootstrap starting.
I20260620 02:40:40.048924 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:40.029870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.0.97.62:36749)
0620 02:40:40.031130 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:40.031149 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:40.031212 (+ 63us) server_negotiation.cc:407] Connection header received
0620 02:40:40.034202 (+ 2990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:40.034237 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:40.034875 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:40.035421 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:40.036400 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:40.038606 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:40.041049 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:40.042253 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:40.042827 (+ 574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:40.044874 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:40.044891 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:40.044925 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:40.046130 (+ 1205us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:40.046296 (+ 166us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:40.046346 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:40.046560 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:40.046878 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 02:40:40.047563 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":485,"threads_started":1}
I20260620 02:40:40.051129 18191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:40.052872 18191 log.cc:824] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:40.058383 18191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b: No bootstrap required, opened a new log
I20260620 02:40:40.067366 18191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed2ffebc7b4977a87be7dc8712ca1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } }
I20260620 02:40:40.068030 18191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:40.068164 18191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fed2ffebc7b4977a87be7dc8712ca1b, State: Initialized, Role: FOLLOWER
I20260620 02:40:40.068964 18191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed2ffebc7b4977a87be7dc8712ca1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } }
I20260620 02:40:40.069267 18191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:40:40.069440 18191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:40:40.069597 18191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:40.072700 18191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed2ffebc7b4977a87be7dc8712ca1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } }
I20260620 02:40:40.073621 18191 leader_election.cc:302] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fed2ffebc7b4977a87be7dc8712ca1b; no voters:
I20260620 02:40:40.074508 18191 leader_election.cc:288] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:40:40.075114 18196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:40:40.076668 18196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [term 1 LEADER]: Becoming Leader. State: Replica: 3fed2ffebc7b4977a87be7dc8712ca1b, State: Running, Role: LEADER
I20260620 02:40:40.077402 18196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed2ffebc7b4977a87be7dc8712ca1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } }
I20260620 02:40:40.077688 18191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:40:40.080606 18198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fed2ffebc7b4977a87be7dc8712ca1b. Latest consensus state: current_term: 1 leader_uuid: "3fed2ffebc7b4977a87be7dc8712ca1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed2ffebc7b4977a87be7dc8712ca1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } } }
I20260620 02:40:40.081054 18198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:40.081665 18197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fed2ffebc7b4977a87be7dc8712ca1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fed2ffebc7b4977a87be7dc8712ca1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36749 } } }
I20260620 02:40:40.082037 18197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:40.082157 18205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:40:40.091374 18205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:40:40.106256 18205 catalog_manager.cc:1347] Generated new cluster ID: e0a2a239f6374eac91eee6f3d606b5d5
I20260620 02:40:40.106909 18205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:40:40.119415 18205 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:40:40.120975 18205 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:40:40.136251 18205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b: Generated new TSK 0
I20260620 02:40:40.137293 18205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:40:40.145495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--builtin_ntp_servers=127.0.97.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:40:40.605945 18215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:40.606290 18215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:40.606498 18215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:40.615269 18215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:40.615429 18215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:40.615660 18215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:40:40.627527 18215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:40.629103 18215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:40.631539 18215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:40.650950 18220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:40.651018 18223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:40.651985 18221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:40.652714 18215 server_base.cc:1034] running on GCE node
I20260620 02:40:40.653857 18215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:40.656088 18215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:40.657510 18215 hybrid_clock.cc:648] HybridClock initialized: now 1781923240657445 us; error 98 us; skew 500 ppm
I20260620 02:40:40.658121 18215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:40:40.662535 18215 webserver.cc:460] Webserver started at http://127.0.97.1:40129/ using document root <none> and password file <none>
I20260620 02:40:40.664104 18215 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:40.664309 18215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:40.664937 18215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:40.669611 18215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "76b4265c996d4e2b9e2e5ccd960f8658"
format_stamp: "Formatted at 2026-06-20 02:40:40 on dist-test-slave-rpg6"
server_key: "68348a40dfa8bc3cbe47541f4269723c"
server_key_iv: "21fcead79628776a3b60997654eb94e5"
server_key_version: "encryptionkey@0"
I20260620 02:40:40.670630 18215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "76b4265c996d4e2b9e2e5ccd960f8658"
format_stamp: "Formatted at 2026-06-20 02:40:40 on dist-test-slave-rpg6"
server_key: "68348a40dfa8bc3cbe47541f4269723c"
server_key_iv: "21fcead79628776a3b60997654eb94e5"
server_key_version: "encryptionkey@0"
I20260620 02:40:40.677163 18215 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:40:40.683035 18229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:40.685796 18215 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 02:40:40.686259 18215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "76b4265c996d4e2b9e2e5ccd960f8658"
format_stamp: "Formatted at 2026-06-20 02:40:40 on dist-test-slave-rpg6"
server_key: "68348a40dfa8bc3cbe47541f4269723c"
server_key_iv: "21fcead79628776a3b60997654eb94e5"
server_key_version: "encryptionkey@0"
I20260620 02:40:40.686616 18215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:40.736604 18215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:40.737618 18215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:40.738240 18215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:40.740099 18215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:40.742591 18215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:40.742851 18215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:40.743045 18215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:40.743150 18215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:40.800347 18341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41945 every 8 connection(s)
I20260620 02:40:40.800354 18215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41945
I20260620 02:40:40.802462 18215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:40:40.808568 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 18215
I20260620 02:40:40.808780 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:40:40.809286 388 external_mini_cluster.cc:1428] Setting key 421ea06af5829616946d7e3568435816
I20260620 02:40:40.813426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--builtin_ntp_servers=127.0.97.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:40:40.824510 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:40.807408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52041 (local address 127.0.97.62:36749)
0620 02:40:40.808031 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:40.808035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:40.809737 (+ 1702us) server_negotiation.cc:407] Connection header received
0620 02:40:40.812548 (+ 2811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:40.812556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:40.812655 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:40.812786 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:40.816487 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:40.817726 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:40.819989 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:40.820784 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:40.820995 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:40.823197 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:40.823207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:40.823211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:40.823250 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:40.823330 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:40.823356 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:40.823463 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:40.823868 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 02:40:40.823997 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":315,"threads_started":1}
I20260620 02:40:40.825670 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:40.808162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36749 (local address 127.0.97.1:52041)
0620 02:40:40.809516 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:40.809571 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:40.812210 (+ 2639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:40.814389 (+ 2179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:40.814403 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:40.814532 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:40.815816 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:40.816201 (+ 385us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:40.818015 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:40.818021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:40.819718 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:40.819739 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:40.821128 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:40.821135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:40.821447 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:40.822718 (+ 1271us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:40:40.822721 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:40.822776 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:40:40.822898 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:40.823536 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:40.823557 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:40.823564 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:40:40.823827 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 02:40:40.824373 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":475,"threads_started":1}
I20260620 02:40:40.827709 18342 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36749
I20260620 02:40:40.828235 18342 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:40.829983 18342 heartbeater.cc:511] Master 127.0.97.62:36749 requested a full tablet report, sending...
I20260620 02:40:40.834609 18156 ts_manager.cc:194] Registered new tserver with Master: 76b4265c996d4e2b9e2e5ccd960f8658 (127.0.97.1:41945)
I20260620 02:40:40.837268 18156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52041
W20260620 02:40:41.276014 18346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:41.276423 18346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:41.276585 18346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:41.284991 18346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:41.285188 18346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:41.285411 18346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:40:41.295985 18346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:41.297444 18346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:41.300433 18346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:41.319620 18354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:41.319650 18351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:41.319653 18352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:41.321131 18346 server_base.cc:1034] running on GCE node
I20260620 02:40:41.322014 18346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:41.323293 18346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:41.325090 18346 hybrid_clock.cc:648] HybridClock initialized: now 1781923241324817 us; error 94 us; skew 500 ppm
I20260620 02:40:41.325532 18346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:40:41.328982 18346 webserver.cc:460] Webserver started at http://127.0.97.2:40435/ using document root <none> and password file <none>
I20260620 02:40:41.329879 18346 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:41.329988 18346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:41.330391 18346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:41.333817 18346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "7c4aef8f1e024158a28e32d29c4df4fe"
format_stamp: "Formatted at 2026-06-20 02:40:41 on dist-test-slave-rpg6"
server_key: "c298d5ecfbcd4ac6832dd9b499340fd6"
server_key_iv: "0b37e3ef3cc92d31a53b258b39e5e18c"
server_key_version: "encryptionkey@0"
I20260620 02:40:41.334757 18346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7c4aef8f1e024158a28e32d29c4df4fe"
format_stamp: "Formatted at 2026-06-20 02:40:41 on dist-test-slave-rpg6"
server_key: "c298d5ecfbcd4ac6832dd9b499340fd6"
server_key_iv: "0b37e3ef3cc92d31a53b258b39e5e18c"
server_key_version: "encryptionkey@0"
I20260620 02:40:41.341293 18346 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:40:41.346711 18360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:41.349059 18346 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:40:41.349296 18346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "7c4aef8f1e024158a28e32d29c4df4fe"
format_stamp: "Formatted at 2026-06-20 02:40:41 on dist-test-slave-rpg6"
server_key: "c298d5ecfbcd4ac6832dd9b499340fd6"
server_key_iv: "0b37e3ef3cc92d31a53b258b39e5e18c"
server_key_version: "encryptionkey@0"
I20260620 02:40:41.349598 18346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:41.381924 18346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:41.383086 18346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:41.383440 18346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:41.384855 18346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:41.387615 18346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:41.387768 18346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:41.388128 18346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:41.388265 18346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:41.430122 18346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40401
I20260620 02:40:41.430183 18472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40401 every 8 connection(s)
I20260620 02:40:41.431686 18346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:40:41.433149 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 18346
I20260620 02:40:41.433351 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:40:41.434012 388 external_mini_cluster.cc:1428] Setting key e8b2ffc6d1e760eca907f39eb31e25fc
I20260620 02:40:41.438655 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--builtin_ntp_servers=127.0.97.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:40:41.451442 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:41.436034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59009 (local address 127.0.97.62:36749)
0620 02:40:41.436645 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:41.436651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:41.437974 (+ 1323us) server_negotiation.cc:407] Connection header received
0620 02:40:41.441473 (+ 3499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:41.441478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:41.441563 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:41.441761 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:41.443876 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:41.444812 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:41.447666 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:41.448258 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:41.448439 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:41.450232 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:41.450241 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:41.450249 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:41.450280 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:41.450339 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:41.450358 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:41.450464 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:41.450797 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 02:40:41.450941 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":355,"threads_started":1}
I20260620 02:40:41.452731 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:41.436487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36749 (local address 127.0.97.2:59009)
0620 02:40:41.437769 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:41.437833 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:41.441025 (+ 3192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:41.442036 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:41.442052 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:41.442195 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:41.443531 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:41.443565 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:41.445016 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:41.445025 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:41.447424 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:41.447457 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:41.448600 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:41.448606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:41.448899 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:41.449834 (+ 935us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:40:41.449837 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:41.449884 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:40:41.449981 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:41.450516 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:41.450526 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:41.450535 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:40:41.450768 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 02:40:41.451178 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":360,"threads_started":1}
I20260620 02:40:41.455427 18473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36749
I20260620 02:40:41.455909 18473 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:41.456786 18473 heartbeater.cc:511] Master 127.0.97.62:36749 requested a full tablet report, sending...
I20260620 02:40:41.458663 18156 ts_manager.cc:194] Registered new tserver with Master: 7c4aef8f1e024158a28e32d29c4df4fe (127.0.97.2:40401)
I20260620 02:40:41.459998 18156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:59009
W20260620 02:40:41.834412 18477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:41.835189 18477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:41.835333 18477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 02:40:41.842160 18342 heartbeater.cc:503] Master 127.0.97.62:36749 was elected leader, sending a full tablet report...
W20260620 02:40:41.847023 18477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:41.847167 18477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:41.847400 18477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:40:41.859453 18477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:41.860981 18477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:41.863054 18477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:41.881454 18482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:41.881500 18483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:41.881520 18485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:41.884639 18477 server_base.cc:1034] running on GCE node
I20260620 02:40:41.885592 18477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:41.886844 18477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:41.888132 18477 hybrid_clock.cc:648] HybridClock initialized: now 1781923241888077 us; error 40 us; skew 500 ppm
I20260620 02:40:41.888607 18477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:40:41.892031 18477 webserver.cc:460] Webserver started at http://127.0.97.3:34801/ using document root <none> and password file <none>
I20260620 02:40:41.892917 18477 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:41.893040 18477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:41.893582 18477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:41.897343 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "2aa22ed110e543af90a7d7fee96bd602"
format_stamp: "Formatted at 2026-06-20 02:40:41 on dist-test-slave-rpg6"
server_key: "ed4c79fb342ba927522e8c2e9e7e803b"
server_key_iv: "e70f53cbbdec2f2baf9ae4c037d60f66"
server_key_version: "encryptionkey@0"
I20260620 02:40:41.898295 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2aa22ed110e543af90a7d7fee96bd602"
format_stamp: "Formatted at 2026-06-20 02:40:41 on dist-test-slave-rpg6"
server_key: "ed4c79fb342ba927522e8c2e9e7e803b"
server_key_iv: "e70f53cbbdec2f2baf9ae4c037d60f66"
server_key_version: "encryptionkey@0"
I20260620 02:40:41.904675 18477 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 02:40:41.909619 18491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:41.911933 18477 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:40:41.912182 18477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "2aa22ed110e543af90a7d7fee96bd602"
format_stamp: "Formatted at 2026-06-20 02:40:41 on dist-test-slave-rpg6"
server_key: "ed4c79fb342ba927522e8c2e9e7e803b"
server_key_iv: "e70f53cbbdec2f2baf9ae4c037d60f66"
server_key_version: "encryptionkey@0"
I20260620 02:40:41.912539 18477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:41.937700 18477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:41.938715 18477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:41.939064 18477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:41.940537 18477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:41.942441 18477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:41.942541 18477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:41.942625 18477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:41.942682 18477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:41.993518 18477 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37427
I20260620 02:40:41.993554 18603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37427 every 8 connection(s)
I20260620 02:40:41.995042 18477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:40:41.996109 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 18477
I20260620 02:40:41.996291 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:40:41.996963 388 external_mini_cluster.cc:1428] Setting key c76653d11e01830d7804a604b454aa11
I20260620 02:40:42.011853 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:41.998657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35089 (local address 127.0.97.62:36749)
0620 02:40:41.999168 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:41.999174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:42.001258 (+ 2084us) server_negotiation.cc:407] Connection header received
0620 02:40:42.003386 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:42.003397 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:42.003501 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:42.003677 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:42.005021 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.005807 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.007581 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.008405 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.008596 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.010699 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:42.010706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:42.010710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:42.010739 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:42.010798 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:42.010815 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:42.010936 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:42.011316 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 02:40:42.011439 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":277,"threads_started":1}
I20260620 02:40:42.013281 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:41.999546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36749 (local address 127.0.97.3:35089)
0620 02:40:42.001083 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.001133 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:42.003055 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:42.004016 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:42.004027 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:42.004137 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:42.004807 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.004829 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.006016 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.006020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.007388 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.007404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.008713 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.008721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.009046 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.010209 (+ 1163us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:40:42.010212 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:42.010267 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:40:42.010374 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:42.010938 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:42.010953 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:42.010964 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:40:42.011192 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 02:40:42.011699 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":751,"threads_started":1}
I20260620 02:40:42.016004 18604 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36749
I20260620 02:40:42.016399 18604 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:42.017567 18604 heartbeater.cc:511] Master 127.0.97.62:36749 requested a full tablet report, sending...
I20260620 02:40:42.019440 18155 ts_manager.cc:194] Registered new tserver with Master: 2aa22ed110e543af90a7d7fee96bd602 (127.0.97.3:37427)
I20260620 02:40:42.020978 18155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:35089
I20260620 02:40:42.029717 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:40:42.046200 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.035186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.0.97.62:36749)
0620 02:40:42.035479 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:42.035484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:42.036346 (+ 862us) server_negotiation.cc:407] Connection header received
0620 02:40:42.036675 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:42.036681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:42.036799 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:42.036924 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:42.037999 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.038815 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.042344 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.043168 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.043344 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.045266 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:42.045272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:42.045276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:42.045305 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:42.045342 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:42.045355 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:42.045452 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:42.045806 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 02:40:42.045923 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260620 02:40:42.052755 18155 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:40:42.061951 18155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:40:42.096728 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.080400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.0.97.3:37427)
0620 02:40:42.085330 (+ 4930us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:42.085342 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:42.086742 (+ 1400us) server_negotiation.cc:407] Connection header received
0620 02:40:42.086943 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:42.086952 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:42.087264 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:42.087504 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:42.088473 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.089655 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.093734 (+ 4079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.094732 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.094927 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.095247 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:42.096108 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 02:40:42.096318 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4751,"thread_start_us":4603,"threads_started":1}
I20260620 02:40:42.097286 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.079902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40401 (local address 127.0.0.1:50978)
0620 02:40:42.084293 (+ 4391us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.084336 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:42.084562 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:42.086582 (+ 2020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:42.086592 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:42.086628 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:42.087311 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.087327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.089423 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.089429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.093079 (+ 3650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.093101 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.096317 (+ 3216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.096324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.096507 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.096541 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:40:42.096633 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 02:40:42.096783 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4076,"mutex_wait_us":154,"thread_start_us":5881,"threads_started":1}
I20260620 02:40:42.097765 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.080963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37427 (local address 127.0.0.1:48358)
0620 02:40:42.086599 (+ 5636us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.086617 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:42.086765 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:42.087806 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:42.087813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:42.087873 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:42.088285 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.088294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.089869 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.089874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.093106 (+ 3232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.093122 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.094978 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.094985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.095238 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.095311 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 02:40:42.095442 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 02:40:42.095670 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5482,"mutex_wait_us":38,"thread_start_us":2813,"threads_started":1}
I20260620 02:40:42.097815 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.080716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41945 (local address 127.0.0.1:56232)
0620 02:40:42.086085 (+ 5369us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.086103 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:42.086248 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:42.087523 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:42.087527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:42.087553 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:42.088010 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.088022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.089866 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.089872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.093498 (+ 3626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.093509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.096913 (+ 3404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.096919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.097105 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.097145 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:40:42.097242 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 02:40:42.097406 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5202,"mutex_wait_us":666,"thread_start_us":4818,"threads_started":1}
I20260620 02:40:42.097936 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.079497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.0.97.2:40401)
0620 02:40:42.081738 (+ 2241us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:42.081750 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:42.084528 (+ 2778us) server_negotiation.cc:407] Connection header received
0620 02:40:42.084893 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:42.084908 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:42.085507 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:42.086051 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:42.087533 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.089246 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.093395 (+ 4149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.095203 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.095352 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.096162 (+ 810us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:42.097333 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0620 02:40:42.097542 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1706,"spinlock_wait_cycles":40704,"thread_start_us":1838,"threads_started":1}
I20260620 02:40:42.098620 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.082133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.0.97.1:41945)
0620 02:40:42.083225 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:42.083244 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:42.086257 (+ 3013us) server_negotiation.cc:407] Connection header received
0620 02:40:42.087026 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:42.087037 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:42.087303 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:42.087499 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:42.088206 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.089664 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.093711 (+ 4047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.096322 (+ 2611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.096473 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.096750 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:42.097623 (+ 873us) server_negotiation.cc:299] Negotiation successful
0620 02:40:42.097818 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":1270,"threads_started":1}
I20260620 02:40:42.103467 18277 tablet_service.cc:1467] Processing CreateTablet for tablet 82b864b1028d475e9a527842d0d9b700 (DEFAULT_TABLE table=test-table [id=245b798dbea24420b1d46ee7ab8d547e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:42.103302 18408 tablet_service.cc:1467] Processing CreateTablet for tablet 82b864b1028d475e9a527842d0d9b700 (DEFAULT_TABLE table=test-table [id=245b798dbea24420b1d46ee7ab8d547e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:42.103303 18539 tablet_service.cc:1467] Processing CreateTablet for tablet 82b864b1028d475e9a527842d0d9b700 (DEFAULT_TABLE table=test-table [id=245b798dbea24420b1d46ee7ab8d547e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:42.105214 18277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b864b1028d475e9a527842d0d9b700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:42.105230 18539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b864b1028d475e9a527842d0d9b700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:42.105230 18408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b864b1028d475e9a527842d0d9b700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:42.121001 18623 tablet_bootstrap.cc:492] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658: Bootstrap starting.
I20260620 02:40:42.122135 18625 tablet_bootstrap.cc:492] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: Bootstrap starting.
I20260620 02:40:42.126019 18625 tablet_bootstrap.cc:654] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:42.126618 18624 tablet_bootstrap.cc:492] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: Bootstrap starting.
I20260620 02:40:42.127466 18625 log.cc:824] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:42.127669 18623 tablet_bootstrap.cc:654] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:42.128930 18623 log.cc:824] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:42.130671 18624 tablet_bootstrap.cc:654] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:42.131156 18625 tablet_bootstrap.cc:492] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: No bootstrap required, opened a new log
I20260620 02:40:42.131659 18625 ts_tablet_manager.cc:1397] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 02:40:42.131983 18624 log.cc:824] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:42.132449 18623 tablet_bootstrap.cc:492] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658: No bootstrap required, opened a new log
I20260620 02:40:42.132746 18623 ts_tablet_manager.cc:1397] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 02:40:42.139987 18624 tablet_bootstrap.cc:492] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: No bootstrap required, opened a new log
I20260620 02:40:42.140941 18624 ts_tablet_manager.cc:1397] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260620 02:40:42.143343 18625 raft_consensus.cc:348] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40401 } } peers { permanent_uuid: "2aa22ed110e543af90a7d7fee96bd602" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37427 } } peers { permanent_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41945 } }
I20260620 02:40:42.143913 18625 raft_consensus.cc:374] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:42.144033 18625 raft_consensus.cc:729] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4aef8f1e024158a28e32d29c4df4fe, State: Initialized, Role: FOLLOWER
I20260620 02:40:42.144377 18623 raft_consensus.cc:348] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40401 } } peers { permanent_uuid: "2aa22ed110e543af90a7d7fee96bd602" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37427 } } peers { permanent_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41945 } }
I20260620 02:40:42.144850 18623 raft_consensus.cc:374] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:42.144960 18623 raft_consensus.cc:729] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b4265c996d4e2b9e2e5ccd960f8658, State: Initialized, Role: FOLLOWER
I20260620 02:40:42.144842 18625 consensus_queue.cc:260] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40401 } } peers { permanent_uuid: "2aa22ed110e543af90a7d7fee96bd602" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37427 } } peers { permanent_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41945 } }
I20260620 02:40:42.145736 18623 consensus_queue.cc:260] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40401 } } peers { permanent_uuid: "2aa22ed110e543af90a7d7fee96bd602" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37427 } } peers { permanent_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41945 } }
I20260620 02:40:42.146587 18473 heartbeater.cc:503] Master 127.0.97.62:36749 was elected leader, sending a full tablet report...
I20260620 02:40:42.147109 18625 ts_tablet_manager.cc:1428] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: Time spent starting tablet: real 0.015s user 0.009s sys 0.008s
I20260620 02:40:42.148715 18623 ts_tablet_manager.cc:1428] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260620 02:40:42.153507 18624 raft_consensus.cc:348] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40401 } } peers { permanent_uuid: "2aa22ed110e543af90a7d7fee96bd602" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37427 } } peers { permanent_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41945 } }
I20260620 02:40:42.154045 18624 raft_consensus.cc:374] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:42.154157 18624 raft_consensus.cc:729] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa22ed110e543af90a7d7fee96bd602, State: Initialized, Role: FOLLOWER
I20260620 02:40:42.155012 18624 consensus_queue.cc:260] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40401 } } peers { permanent_uuid: "2aa22ed110e543af90a7d7fee96bd602" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37427 } } peers { permanent_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41945 } }
I20260620 02:40:42.156342 18604 heartbeater.cc:503] Master 127.0.97.62:36749 was elected leader, sending a full tablet report...
I20260620 02:40:42.157045 18624 ts_tablet_manager.cc:1428] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260620 02:40:42.181126 18630 raft_consensus.cc:482] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:40:42.181439 18630 raft_consensus.cc:504] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40401 } } peers { permanent_uuid: "2aa22ed110e543af90a7d7fee96bd602" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37427 } } peers { permanent_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41945 } }
I20260620 02:40:42.183413 18630 leader_election.cc:288] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c4aef8f1e024158a28e32d29c4df4fe (127.0.97.2:40401), 2aa22ed110e543af90a7d7fee96bd602 (127.0.97.3:37427)
W20260620 02:40:42.184645 18474 tablet.cc:2369] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:40:42.194717 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.183950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33581 (local address 127.0.97.2:40401)
0620 02:40:42.184294 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:42.184304 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:42.185976 (+ 1672us) server_negotiation.cc:407] Connection header received
0620 02:40:42.186243 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:42.186252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:42.186363 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:42.186506 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:42.187210 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.188286 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.191617 (+ 3331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.193044 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.193204 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.193327 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:42.194274 (+ 947us) server_negotiation.cc:299] Negotiation successful
0620 02:40:42.194432 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:40:42.194718 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.183980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54017 (local address 127.0.97.3:37427)
0620 02:40:42.184200 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:42.184209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:42.186104 (+ 1895us) server_negotiation.cc:407] Connection header received
0620 02:40:42.186360 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:42.186367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:42.186477 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:42.186589 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:42.187428 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.188398 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.191617 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.193038 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.193204 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.193364 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:42.194274 (+ 910us) server_negotiation.cc:299] Negotiation successful
0620 02:40:42.194473 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:40:42.194832 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.183985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37427 (local address 127.0.97.1:54017)
0620 02:40:42.185950 (+ 1965us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.185970 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:42.186144 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:42.186750 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:42.186758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:42.186787 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:42.187225 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.187237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.188573 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.188578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.191310 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.191333 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.193378 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.193383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.193885 (+ 502us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.194033 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 02:40:42.194156 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 02:40:42.194345 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1820,"mutex_wait_us":300,"thread_start_us":1140,"threads_started":1}
I20260620 02:40:42.195076 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.183968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40401 (local address 127.0.97.1:33581)
0620 02:40:42.185798 (+ 1830us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.185817 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:42.186059 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:42.186511 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:42.186519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:42.186550 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:42.187022 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.187037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.188437 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.188446 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.191310 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:42.191333 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.193365 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.193374 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:42.193887 (+ 513us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.194033 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 02:40:42.194266 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 02:40:42.194414 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1700,"mutex_wait_us":508,"thread_start_us":1351,"threads_started":1}
I20260620 02:40:42.196090 18559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b864b1028d475e9a527842d0d9b700" candidate_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa22ed110e543af90a7d7fee96bd602" is_pre_election: true
I20260620 02:40:42.196444 18428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b864b1028d475e9a527842d0d9b700" candidate_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" is_pre_election: true
I20260620 02:40:42.196645 18559 raft_consensus.cc:2454] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b4265c996d4e2b9e2e5ccd960f8658 in term 0.
I20260620 02:40:42.197083 18428 raft_consensus.cc:2454] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b4265c996d4e2b9e2e5ccd960f8658 in term 0.
I20260620 02:40:42.198050 18231 leader_election.cc:302] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa22ed110e543af90a7d7fee96bd602, 76b4265c996d4e2b9e2e5ccd960f8658; no voters:
I20260620 02:40:42.198587 18630 raft_consensus.cc:2781] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:40:42.198762 18630 raft_consensus.cc:482] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:40:42.198882 18630 raft_consensus.cc:3037] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:42.201388 18630 raft_consensus.cc:504] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40401 } } peers { permanent_uuid: "2aa22ed110e543af90a7d7fee96bd602" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37427 } } peers { permanent_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41945 } }
I20260620 02:40:42.202109 18630 leader_election.cc:288] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [CANDIDATE]: Term 1 election: Requested vote from peers 7c4aef8f1e024158a28e32d29c4df4fe (127.0.97.2:40401), 2aa22ed110e543af90a7d7fee96bd602 (127.0.97.3:37427)
I20260620 02:40:42.202805 18428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b864b1028d475e9a527842d0d9b700" candidate_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4aef8f1e024158a28e32d29c4df4fe"
I20260620 02:40:42.202801 18559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b864b1028d475e9a527842d0d9b700" candidate_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa22ed110e543af90a7d7fee96bd602"
I20260620 02:40:42.203076 18428 raft_consensus.cc:3037] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:42.203091 18559 raft_consensus.cc:3037] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:42.205739 18559 raft_consensus.cc:2454] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b4265c996d4e2b9e2e5ccd960f8658 in term 1.
I20260620 02:40:42.205766 18428 raft_consensus.cc:2454] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b4265c996d4e2b9e2e5ccd960f8658 in term 1.
I20260620 02:40:42.206377 18231 leader_election.cc:302] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2aa22ed110e543af90a7d7fee96bd602, 76b4265c996d4e2b9e2e5ccd960f8658, 7c4aef8f1e024158a28e32d29c4df4fe; no voters:
I20260620 02:40:42.206818 18630 raft_consensus.cc:2781] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:40:42.207391 18630 raft_consensus.cc:686] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 1 LEADER]: Becoming Leader. State: Replica: 76b4265c996d4e2b9e2e5ccd960f8658, State: Running, Role: LEADER
I20260620 02:40:42.208132 18630 consensus_queue.cc:237] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40401 } } peers { permanent_uuid: "2aa22ed110e543af90a7d7fee96bd602" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37427 } } peers { permanent_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41945 } }
I20260620 02:40:42.213385 18155 catalog_manager.cc:5495] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76b4265c996d4e2b9e2e5ccd960f8658 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa22ed110e543af90a7d7fee96bd602" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76b4265c996d4e2b9e2e5ccd960f8658" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41945 } health_report { overall_health: HEALTHY } } }
W20260620 02:40:42.247498 18605 tablet.cc:2369] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:40:42.261821 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.253898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.0.97.1:41945)
0620 02:40:42.254125 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:42.254129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:42.254147 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:40:42.254248 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:42.254252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:42.254353 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:42.254497 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:40:42.255231 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.256392 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.258422 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.259033 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.259165 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.260390 (+ 1225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:40:42.261059 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:40:42.261236 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:42.261502 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 02:40:42.261589 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:40:42.272491 18559 raft_consensus.cc:1261] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602 [term 1 FOLLOWER]: Refusing update from remote peer 76b4265c996d4e2b9e2e5ccd960f8658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:40:42.272578 18428 raft_consensus.cc:1261] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe [term 1 FOLLOWER]: Refusing update from remote peer 76b4265c996d4e2b9e2e5ccd960f8658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:40:42.273892 18630 consensus_queue.cc:1035] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa22ed110e543af90a7d7fee96bd602" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:40:42.274394 18634 consensus_queue.cc:1035] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c4aef8f1e024158a28e32d29c4df4fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:40:42.293903 18637 mvcc.cc:205] Tried to move back new op lower bound from 7298757600334860288 to 7298757600088829952. Current Snapshot: MvccSnapshot[applied={T|T < 7298757600334860288}]
I20260620 02:40:42.305693 18638 mvcc.cc:205] Tried to move back new op lower bound from 7298757600334860288 to 7298757600088829952. Current Snapshot: MvccSnapshot[applied={T|T < 7298757600334860288 or (T in {7298757600334860288})}]
I20260620 02:40:42.320727 18155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:40:42.321079 18155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:40:42.325789 18155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fed2ffebc7b4977a87be7dc8712ca1b: Sending DeleteTablet for 3 replicas of tablet 82b864b1028d475e9a527842d0d9b700
I20260620 02:40:42.327164 18539 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b864b1028d475e9a527842d0d9b700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:42 UTC) from {username='slave'} at 127.0.0.1:48358
I20260620 02:40:42.327152 18408 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b864b1028d475e9a527842d0d9b700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:42 UTC) from {username='slave'} at 127.0.0.1:50978
I20260620 02:40:42.327742 18277 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b864b1028d475e9a527842d0d9b700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:42 UTC) from {username='slave'} at 127.0.0.1:56232
I20260620 02:40:42.328728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 18215
I20260620 02:40:42.340303 18648 tablet_replica.cc:330] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: stopping tablet replica
I20260620 02:40:42.341212 18647 tablet_replica.cc:330] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: stopping tablet replica
I20260620 02:40:42.342145 18647 raft_consensus.cc:2229] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:40:42.342846 18648 raft_consensus.cc:2229] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:40:42.343706 18647 raft_consensus.cc:2258] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:42.344383 18648 raft_consensus.cc:2258] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:42.345806 18649 tablet_replica.cc:330] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658: stopping tablet replica
I20260620 02:40:42.346329 18649 raft_consensus.cc:2229] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:40:42.347159 18649 raft_consensus.cc:2258] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:42.349403 18647 ts_tablet_manager.cc:1905] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:42.349713 18648 ts_tablet_manager.cc:1905] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:42.351001 18649 ts_tablet_manager.cc:1905] T 82b864b1028d475e9a527842d0d9b700 P 76b4265c996d4e2b9e2e5ccd960f8658: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:42.354352 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.330640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.0.97.1:41945)
0620 02:40:42.330892 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:42.330902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:42.342346 (+ 11444us) server_negotiation.cc:407] Connection header received
0620 02:40:42.343052 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:42.343058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:42.343183 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:42.343270 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:42.344063 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.345428 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.348430 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.349125 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.349263 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.352052 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:42.352069 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:42.352093 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:42.353104 (+ 1011us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:42.353212 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:42.353244 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:42.353435 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:42.353772 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 02:40:42.353922 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:40:42.355885 18337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:42.359777 18648 ts_tablet_manager.cc:1918] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:42.360244 18648 log.cc:1196] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/82b864b1028d475e9a527842d0d9b700
I20260620 02:40:42.360832 18648 ts_tablet_manager.cc:1939] T 82b864b1028d475e9a527842d0d9b700 P 7c4aef8f1e024158a28e32d29c4df4fe: Deleting consensus metadata
I20260620 02:40:42.362802 18142 catalog_manager.cc:4841] TS 7c4aef8f1e024158a28e32d29c4df4fe (127.0.97.2:40401): tablet 82b864b1028d475e9a527842d0d9b700 (table test-table [id=245b798dbea24420b1d46ee7ab8d547e]) successfully deleted
I20260620 02:40:42.365223 18647 ts_tablet_manager.cc:1918] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:42.365446 18647 log.cc:1196] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/82b864b1028d475e9a527842d0d9b700
I20260620 02:40:42.366086 18647 ts_tablet_manager.cc:1939] T 82b864b1028d475e9a527842d0d9b700 P 2aa22ed110e543af90a7d7fee96bd602: Deleting consensus metadata
I20260620 02:40:42.368463 18141 catalog_manager.cc:4841] TS 2aa22ed110e543af90a7d7fee96bd602 (127.0.97.3:37427): tablet 82b864b1028d475e9a527842d0d9b700 (table test-table [id=245b798dbea24420b1d46ee7ab8d547e]) successfully deleted
I20260620 02:40:42.463742 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 18215
I20260620 02:40:42.484495 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 18346
W20260620 02:40:42.484709 18140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41945 (error 108)
I20260620 02:40:42.487020 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.486590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41945 (local address 127.0.0.1:56248)
0620 02:40:42.486764 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.486867 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 02:40:42.487304 18140 catalog_manager.cc:4584] TS 76b4265c996d4e2b9e2e5ccd960f8658 (127.0.97.1:41945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82b864b1028d475e9a527842d0d9b700: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41945: connect: Connection refused (error 111)
I20260620 02:40:42.492986 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.485811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.0.97.2:40401)
0620 02:40:42.486020 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:42.486026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:42.486047 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:40:42.486111 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:42.486114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:42.486211 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:42.486337 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:42.486775 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.487721 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.489032 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.489457 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.489603 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.490943 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:42.490952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:42.490967 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:42.491664 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:42.491728 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:42.491756 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:42.492339 (+ 583us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:42.492467 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 02:40:42.492607 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:40:42.493885 18468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:42.541364 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.540719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41945 (local address 127.0.0.1:56256)
0620 02:40:42.541003 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.541138 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 02:40:42.543016 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.542536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41945 (local address 127.0.0.1:56264)
0620 02:40:42.542725 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.542834 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 02:40:42.596295 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 18346
I20260620 02:40:42.604637 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.603760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41945 (local address 127.0.0.1:56272)
0620 02:40:42.604246 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.604402 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266}
I20260620 02:40:42.606184 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.605696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41945 (local address 127.0.0.1:56276)
0620 02:40:42.605900 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.606006 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 02:40:42.616351 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 18477
I20260620 02:40:42.626583 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.617766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.0.97.3:37427)
0620 02:40:42.618060 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:42.618067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:42.618103 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:40:42.618335 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:42.618341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:42.618444 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:42.618540 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:42.619085 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.620323 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.622307 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:42.622772 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:42.622912 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:42.624980 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:42.624991 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:42.625006 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:42.625626 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:42.625692 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:42.625716 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:42.625882 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:42.626167 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 02:40:42.626302 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 02:40:42.628321 18599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:42.687804 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.687252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41945 (local address 127.0.0.1:56282)
0620 02:40:42.687500 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.687600 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 02:40:42.689025 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.688630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41945 (local address 127.0.0.1:56296)
0620 02:40:42.688831 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.688913 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 02:40:42.737744 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 18477
I20260620 02:40:42.756582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 18125
I20260620 02:40:42.758646 18186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:42.882930 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:42.882097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41945 (local address 127.0.0.1:56304)
0620 02:40:42.882306 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:42.882440 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 02:40:42.883898 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 18125
2026-06-20T02:40:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3567 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:40:43 dist-test-slave-rpg6 krb5kdc[18658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:40:43 dist-test-slave-rpg6 krb5kdc[18658](info): set up 2 sockets
Jun 20 02:40:43 dist-test-slave-rpg6 krb5kdc[18658](info): commencing operation
krb5kdc: starting...
W20260620 02:40:44.141579 388 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:40:44 dist-test-slave-rpg6 krb5kdc[18658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:40:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:40:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:40:44.456867 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39471
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:40:44.892495 18671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:44.893067 18671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:44.893262 18671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:44.904282 18671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:40:44.904469 18671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:44.904580 18671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:44.904692 18671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:40:44.904769 18671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:40:44.915894 18671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:44.917725 18671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:44.919363 18671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:44.934516 18676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:44.935257 18677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:44.935264 18679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:44.936290 18671 server_base.cc:1034] running on GCE node
I20260620 02:40:44.938684 18671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:44.941674 18671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:44.943176 18671 hybrid_clock.cc:648] HybridClock initialized: now 1781923244943097 us; error 86 us; skew 500 ppm
I20260620 02:40:44.943994 18671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:44 dist-test-slave-rpg6 krb5kdc[18658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:44.956259 18671 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:40:44.958952 18671 webserver.cc:460] Webserver started at http://127.0.97.62:37411/ using document root <none> and password file <none>
I20260620 02:40:44.959806 18671 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:44.960001 18671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:44.960309 18671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:44.963743 18671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "e3bcbf4350a04766af2bdf3694778193"
format_stamp: "Formatted at 2026-06-20 02:40:44 on dist-test-slave-rpg6"
server_key: "c12f7557318a9c546b8e32ce94345a3b"
server_key_iv: "850be0b22c12f2082190c778b6354574"
server_key_version: "encryptionkey@0"
I20260620 02:40:44.964764 18671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "e3bcbf4350a04766af2bdf3694778193"
format_stamp: "Formatted at 2026-06-20 02:40:44 on dist-test-slave-rpg6"
server_key: "c12f7557318a9c546b8e32ce94345a3b"
server_key_iv: "850be0b22c12f2082190c778b6354574"
server_key_version: "encryptionkey@0"
I20260620 02:40:44.972491 18671 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 02:40:44.977293 18686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:44.979342 18671 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:40:44.979543 18671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "e3bcbf4350a04766af2bdf3694778193"
format_stamp: "Formatted at 2026-06-20 02:40:44 on dist-test-slave-rpg6"
server_key: "c12f7557318a9c546b8e32ce94345a3b"
server_key_iv: "850be0b22c12f2082190c778b6354574"
server_key_version: "encryptionkey@0"
I20260620 02:40:44.979748 18671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:45.002308 18671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:45.009353 18671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:45.009639 18671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:45.036237 18671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39471
I20260620 02:40:45.036268 18737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39471 every 8 connection(s)
I20260620 02:40:45.037928 18671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:40:45.041159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 18671
I20260620 02:40:45.041407 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:40:45.042101 388 external_mini_cluster.cc:1428] Setting key eb055f7d1ba0b67e41a418e4be1e7011
I20260620 02:40:45.043725 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:45.061596 18738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193: Bootstrap starting.
Jun 20 02:40:45 dist-test-slave-rpg6 krb5kdc[18658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:40:45.069290 18738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:45.071810 18738 log.cc:824] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:45.077281 18738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193: No bootstrap required, opened a new log
I20260620 02:40:45.082074 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:45.045343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.0.97.62:39471)
0620 02:40:45.046231 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:45.046243 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:45.046298 (+ 55us) server_negotiation.cc:407] Connection header received
0620 02:40:45.050024 (+ 3726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:45.050070 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:45.050609 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:45.051195 (+ 586us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:45.053478 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:45.054973 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:45.057160 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:45.058046 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:45.058355 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:45.067238 (+ 8883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:45.067252 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:45.067277 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:45.067356 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:45.074270 (+ 6914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:45.076377 (+ 2107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:45.076386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:45.076400 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:45.076632 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:45.077797 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:45.077810 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:45.077816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:45.078552 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:45.078889 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:45.079749 (+ 860us) server_negotiation.cc:299] Negotiation successful
0620 02:40:45.080376 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":429,"threads_started":1}
I20260620 02:40:45.088600 18738 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bcbf4350a04766af2bdf3694778193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39471 } }
I20260620 02:40:45.089776 18738 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:45.090370 18738 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bcbf4350a04766af2bdf3694778193, State: Initialized, Role: FOLLOWER
I20260620 02:40:45.091521 18738 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bcbf4350a04766af2bdf3694778193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39471 } }
I20260620 02:40:45.091928 18738 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:40:45.092137 18738 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:40:45.092458 18738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:45.095382 18738 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bcbf4350a04766af2bdf3694778193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39471 } }
I20260620 02:40:45.096092 18738 leader_election.cc:302] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3bcbf4350a04766af2bdf3694778193; no voters:
I20260620 02:40:45.097734 18738 leader_election.cc:288] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:40:45.097782 18743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:40:45.099620 18743 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [term 1 LEADER]: Becoming Leader. State: Replica: e3bcbf4350a04766af2bdf3694778193, State: Running, Role: LEADER
I20260620 02:40:45.100932 18738 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:40:45.100764 18743 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bcbf4350a04766af2bdf3694778193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39471 } }
I20260620 02:40:45.111179 18745 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3bcbf4350a04766af2bdf3694778193. Latest consensus state: current_term: 1 leader_uuid: "e3bcbf4350a04766af2bdf3694778193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bcbf4350a04766af2bdf3694778193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39471 } } }
I20260620 02:40:45.111716 18745 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:45.113233 18744 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3bcbf4350a04766af2bdf3694778193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bcbf4350a04766af2bdf3694778193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39471 } } }
I20260620 02:40:45.113739 18744 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193 [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:45.118355 18752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:40:45.132769 18752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:40:45.146767 18752 catalog_manager.cc:1347] Generated new cluster ID: 9c9a8007e36341668bd30d3f900fbc87
I20260620 02:40:45.146952 18752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:40:45.164038 18752 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:40:45.166079 18752 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:40:45.178804 18752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193: Generated new TSK 0
I20260620 02:40:45.179817 18752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:40:45.266501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39471
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:40:45.753755 18764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:45.754463 18764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:45.754630 18764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:45.766335 18764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:45.766512 18764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:45.766644 18764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:40:45.778126 18764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:45.779935 18764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:45.783134 18764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:45.809362 18770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:45.809295 18769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:45.809645 18772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:45.812628 18764 server_base.cc:1034] running on GCE node
I20260620 02:40:45.814556 18764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:45.817051 18764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:45.820597 18764 hybrid_clock.cc:648] HybridClock initialized: now 1781923245820492 us; error 110 us; skew 500 ppm
I20260620 02:40:45.821110 18764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:45 dist-test-slave-rpg6 krb5kdc[18658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:45.827497 18764 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:40:45.831151 18764 webserver.cc:460] Webserver started at http://127.0.97.1:40789/ using document root <none> and password file <none>
I20260620 02:40:45.832260 18764 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:45.832469 18764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:45.832875 18764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:45.837049 18764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "5a6d1ca72d444115bec700e60beaf4f1"
format_stamp: "Formatted at 2026-06-20 02:40:45 on dist-test-slave-rpg6"
server_key: "8dfdfae3edfb0466f33dcc2228534e9f"
server_key_iv: "f9b213dabdb5a342ef90adf2d85f77ba"
server_key_version: "encryptionkey@0"
I20260620 02:40:45.838433 18764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5a6d1ca72d444115bec700e60beaf4f1"
format_stamp: "Formatted at 2026-06-20 02:40:45 on dist-test-slave-rpg6"
server_key: "8dfdfae3edfb0466f33dcc2228534e9f"
server_key_iv: "f9b213dabdb5a342ef90adf2d85f77ba"
server_key_version: "encryptionkey@0"
I20260620 02:40:45.847503 18764 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260620 02:40:45.852855 18779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:45.856393 18764 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:40:45.856745 18764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "5a6d1ca72d444115bec700e60beaf4f1"
format_stamp: "Formatted at 2026-06-20 02:40:45 on dist-test-slave-rpg6"
server_key: "8dfdfae3edfb0466f33dcc2228534e9f"
server_key_iv: "f9b213dabdb5a342ef90adf2d85f77ba"
server_key_version: "encryptionkey@0"
I20260620 02:40:45.857060 18764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:45.882440 18764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:45.891311 18764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:45.892302 18764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:45.894199 18764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:45.896164 18764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:45.896265 18764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:45.896401 18764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:45.896476 18764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:45.960769 18764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35705
I20260620 02:40:45.960780 18891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35705 every 8 connection(s)
I20260620 02:40:45.962620 18764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:40:45.963800 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 18764
I20260620 02:40:45.964054 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:40:45.964615 388 external_mini_cluster.cc:1428] Setting key a7d7d0c9c7d12e4cd917e608027964b5
Jun 20 02:40:45 dist-test-slave-rpg6 krb5kdc[18658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:40:45.996520 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:45.966599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54657 (local address 127.0.97.62:39471)
0620 02:40:45.968296 (+ 1697us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:45.968303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:45.968889 (+ 586us) server_negotiation.cc:407] Connection header received
0620 02:40:45.971085 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:45.971094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:45.971204 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:45.971352 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:45.974445 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:45.975884 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:45.980109 (+ 4225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:45.980622 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:45.980765 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:45.987403 (+ 6638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:45.987411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:45.987414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:45.987471 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:45.991073 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:45.992651 (+ 1578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:45.992658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:45.992661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:45.992857 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:45.993476 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:45.993481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:45.993483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:45.993876 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:45.994449 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:45.995607 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0620 02:40:45.995954 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":1006,"threads_started":1}
I20260620 02:40:45.998102 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:45.967557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39471 (local address 127.0.97.1:54657)
0620 02:40:45.968674 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:45.968736 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:45.970728 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:45.971666 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:45.971676 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:45.973043 (+ 1367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:45.974014 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:45.974048 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:45.976374 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:45.976382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:45.979543 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:45.979582 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:45.980980 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:45.980990 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:45.981438 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:45.982853 (+ 1415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:40:45.982901 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:45.986985 (+ 4084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:45.991395 (+ 4410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:45.991429 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:45.991452 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:45.992378 (+ 926us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:45.993016 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:45.993022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:45.993025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:45.993323 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:45.994233 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:45.994247 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:45.994978 (+ 731us) client_negotiation.cc:769] Sending connection context
0620 02:40:45.995372 (+ 394us) client_negotiation.cc:240] Negotiation successful
0620 02:40:45.996096 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":482,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:40:46.001329 18892 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39471
I20260620 02:40:46.001998 18892 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:46.003021 18892 heartbeater.cc:511] Master 127.0.97.62:39471 requested a full tablet report, sending...
I20260620 02:40:46.007465 18703 ts_manager.cc:194] Registered new tserver with Master: 5a6d1ca72d444115bec700e60beaf4f1 (127.0.97.1:35705)
I20260620 02:40:46.010440 18703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:40:46.046437 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39471
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:40:46.484050 18898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:46.484421 18898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:46.484591 18898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:46.495404 18898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:46.495651 18898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:46.495921 18898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:40:46.509485 18898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:46.511045 18898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:46.512614 18898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:46.534598 18904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:46.535200 18906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:46.536230 18903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:46.537926 18898 server_base.cc:1034] running on GCE node
I20260620 02:40:46.538816 18898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:46.540210 18898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:46.541749 18898 hybrid_clock.cc:648] HybridClock initialized: now 1781923246541555 us; error 194 us; skew 500 ppm
I20260620 02:40:46.542567 18898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:46 dist-test-slave-rpg6 krb5kdc[18658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:46.549563 18898 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:40:46.552054 18898 webserver.cc:460] Webserver started at http://127.0.97.2:39687/ using document root <none> and password file <none>
I20260620 02:40:46.552989 18898 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:46.553117 18898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:46.553486 18898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:46.558662 18898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "d6a8d563291943929335bf59ad0cc51b"
format_stamp: "Formatted at 2026-06-20 02:40:46 on dist-test-slave-rpg6"
server_key: "51565091bd24a75ee79a16336bed25a9"
server_key_iv: "cb78e53aa6245188836cab169a3c98a3"
server_key_version: "encryptionkey@0"
I20260620 02:40:46.560632 18898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d6a8d563291943929335bf59ad0cc51b"
format_stamp: "Formatted at 2026-06-20 02:40:46 on dist-test-slave-rpg6"
server_key: "51565091bd24a75ee79a16336bed25a9"
server_key_iv: "cb78e53aa6245188836cab169a3c98a3"
server_key_version: "encryptionkey@0"
I20260620 02:40:46.569224 18898 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260620 02:40:46.575671 18913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:46.579196 18898 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260620 02:40:46.579505 18898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "d6a8d563291943929335bf59ad0cc51b"
format_stamp: "Formatted at 2026-06-20 02:40:46 on dist-test-slave-rpg6"
server_key: "51565091bd24a75ee79a16336bed25a9"
server_key_iv: "cb78e53aa6245188836cab169a3c98a3"
server_key_version: "encryptionkey@0"
I20260620 02:40:46.579780 18898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:46.621448 18898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:46.628855 18898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:46.629385 18898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:46.631455 18898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:46.634362 18898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:46.634575 18898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:46.634801 18898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:46.634907 18898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:46.696022 18898 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43195
I20260620 02:40:46.696100 19025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43195 every 8 connection(s)
I20260620 02:40:46.698221 18898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:40:46.708775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 18898
I20260620 02:40:46.709012 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:40:46.709488 388 external_mini_cluster.cc:1428] Setting key 7b7c7abb970e8d74cdb03c1941c70f83
Jun 20 02:40:46 dist-test-slave-rpg6 krb5kdc[18658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:40:46.733047 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:46.704406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45913 (local address 127.0.97.62:39471)
0620 02:40:46.706330 (+ 1924us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:46.706338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:46.706365 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:40:46.707991 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:46.707997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:46.708113 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:46.708258 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:46.710551 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:46.712342 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:46.716951 (+ 4609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:46.717519 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:46.717715 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:46.724922 (+ 7207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:46.724931 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:46.724934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:46.725156 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:46.729220 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:46.730280 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:46.730286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:46.730288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:46.730437 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:46.730950 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:46.730960 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:46.730962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:46.731273 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:46.731440 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:46.732369 (+ 929us) server_negotiation.cc:299] Negotiation successful
0620 02:40:46.732661 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1739,"thread_start_us":1571,"threads_started":1}
I20260620 02:40:46.734179 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:46.704385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39471 (local address 127.0.97.2:45913)
0620 02:40:46.705400 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:46.705465 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:46.707583 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:46.708546 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:46.708556 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:46.709545 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:46.710318 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:46.710340 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:46.712955 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:46.712961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:46.716118 (+ 3157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:46.716146 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:46.717874 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:46.717884 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:46.718190 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:46.720021 (+ 1831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:40:46.720075 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:46.724441 (+ 4366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:46.729476 (+ 5035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:46.729491 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:46.729517 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:46.730087 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:46.730580 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:46.730585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:46.730588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:46.730816 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:46.731423 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:46.731432 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:46.731920 (+ 488us) client_negotiation.cc:769] Sending connection context
0620 02:40:46.732260 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 02:40:46.732883 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":447,"threads_started":1}
I20260620 02:40:46.735952 19026 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39471
I20260620 02:40:46.736403 19026 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:46.737466 19026 heartbeater.cc:511] Master 127.0.97.62:39471 requested a full tablet report, sending...
I20260620 02:40:46.740592 18703 ts_manager.cc:194] Registered new tserver with Master: d6a8d563291943929335bf59ad0cc51b (127.0.97.2:43195)
I20260620 02:40:46.741983 18703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45913
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:40:46.794697 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39471
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:40:47.018265 18892 heartbeater.cc:503] Master 127.0.97.62:39471 was elected leader, sending a full tablet report...
W20260620 02:40:47.261723 19032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:47.262094 19032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:47.262260 19032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:47.270274 19032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:47.270416 19032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:47.270637 19032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:40:47.282181 19032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:47.285563 19032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:47.287352 19032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:47.308104 19040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:47.308044 19037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:47.308400 19038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:47.309331 19032 server_base.cc:1034] running on GCE node
I20260620 02:40:47.310770 19032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:47.313295 19032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:47.315111 19032 hybrid_clock.cc:648] HybridClock initialized: now 1781923247315027 us; error 92 us; skew 500 ppm
I20260620 02:40:47.315781 19032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:47 dist-test-slave-rpg6 krb5kdc[18658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:47.320580 19032 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:40:47.322835 19032 webserver.cc:460] Webserver started at http://127.0.97.3:44553/ using document root <none> and password file <none>
I20260620 02:40:47.324064 19032 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:47.324219 19032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:47.324522 19032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:47.328068 19032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "e94c39155e75406288b91fc7299cc85c"
format_stamp: "Formatted at 2026-06-20 02:40:47 on dist-test-slave-rpg6"
server_key: "0c153c5634909e3232d2091b9ebc3699"
server_key_iv: "bc5f40e4a122eb2c2104e01839974003"
server_key_version: "encryptionkey@0"
I20260620 02:40:47.329334 19032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e94c39155e75406288b91fc7299cc85c"
format_stamp: "Formatted at 2026-06-20 02:40:47 on dist-test-slave-rpg6"
server_key: "0c153c5634909e3232d2091b9ebc3699"
server_key_iv: "bc5f40e4a122eb2c2104e01839974003"
server_key_version: "encryptionkey@0"
I20260620 02:40:47.339061 19032 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260620 02:40:47.344744 19047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:47.348786 19032 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:40:47.349128 19032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "e94c39155e75406288b91fc7299cc85c"
format_stamp: "Formatted at 2026-06-20 02:40:47 on dist-test-slave-rpg6"
server_key: "0c153c5634909e3232d2091b9ebc3699"
server_key_iv: "bc5f40e4a122eb2c2104e01839974003"
server_key_version: "encryptionkey@0"
I20260620 02:40:47.349387 19032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:47.370982 19032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:47.380216 19032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:47.380623 19032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:47.381744 19032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:47.383625 19032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:47.383999 19032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 02:40:47.384186 19032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:47.384255 19032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:47.435607 19032 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43945
I20260620 02:40:47.435642 19159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43945 every 8 connection(s)
I20260620 02:40:47.437091 19032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:40:47.444082 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 19032
I20260620 02:40:47.444267 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:40:47.444777 388 external_mini_cluster.cc:1428] Setting key 263f167c1ebab41818f82331b4961cb3
Jun 20 02:40:47 dist-test-slave-rpg6 krb5kdc[18658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:40:47.477296 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.447953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44481 (local address 127.0.97.62:39471)
0620 02:40:47.448557 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:47.448561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:47.448587 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:40:47.450330 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:47.450336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:47.450650 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:47.450778 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:47.453864 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.454974 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.458132 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.458740 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.458917 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.466535 (+ 7618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:47.466544 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:47.466548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:47.466681 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:47.470812 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:47.472496 (+ 1684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:47.472503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:47.472506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:47.472752 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:47.473800 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:47.473807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:47.473913 (+ 106us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:47.474355 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:47.474640 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:47.476473 (+ 1833us) server_negotiation.cc:299] Negotiation successful
0620 02:40:47.476910 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":344,"threads_started":1}
I20260620 02:40:47.479113 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.441230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39471 (local address 127.0.97.3:44481)
0620 02:40:47.447012 (+ 5782us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:47.447081 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:47.449847 (+ 2766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:47.451337 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:47.451352 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:47.452495 (+ 1143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:47.453460 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.453522 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.455250 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.455256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.457739 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.457779 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.459202 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.459214 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.459633 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.461815 (+ 2182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:40:47.461915 (+ 100us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:47.466064 (+ 4149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:47.471131 (+ 5067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:47.471143 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:47.471167 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:47.472158 (+ 991us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:47.473127 (+ 969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:47.473132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:47.473136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:47.473470 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:47.474555 (+ 1085us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:47.474568 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:47.475260 (+ 692us) client_negotiation.cc:769] Sending connection context
0620 02:40:47.476231 (+ 971us) client_negotiation.cc:240] Negotiation successful
0620 02:40:47.477108 (+ 877us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5387,"thread_start_us":5133,"threads_started":1}
I20260620 02:40:47.481483 19160 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39471
I20260620 02:40:47.481863 19160 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:47.482728 19160 heartbeater.cc:511] Master 127.0.97.62:39471 requested a full tablet report, sending...
I20260620 02:40:47.485481 18703 ts_manager.cc:194] Registered new tserver with Master: e94c39155e75406288b91fc7299cc85c (127.0.97.3:43945)
I20260620 02:40:47.487170 18703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44481
I20260620 02:40:47.493943 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:40:47.528352 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.501337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.0.97.62:39471)
0620 02:40:47.501625 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:47.501630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:47.502129 (+ 499us) server_negotiation.cc:407] Connection header received
0620 02:40:47.502257 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:47.502262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:47.502451 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:47.502552 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:47.504702 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.505884 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.509405 (+ 3521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.510237 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.510406 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.516478 (+ 6072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:47.516535 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:47.516553 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:47.516642 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:47.520855 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:47.522770 (+ 1915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:47.522777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:47.522780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:47.523285 (+ 505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:47.524331 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:47.524337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:47.524340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:47.525014 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:47.525354 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:47.526396 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0620 02:40:47.526935 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 02:40:47.537796 18703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:40:47.548130 18703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:40:47.586661 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.569936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.0.97.1:35705)
0620 02:40:47.571847 (+ 1911us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:47.571861 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:47.572648 (+ 787us) server_negotiation.cc:407] Connection header received
0620 02:40:47.572940 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:47.572949 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:47.573535 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:47.573926 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:47.575534 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.577507 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.583230 (+ 5723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.584857 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.585126 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.585447 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:47.586023 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 02:40:47.586229 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1627,"thread_start_us":1456,"threads_started":1}
I20260620 02:40:47.586906 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.569191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43195 (local address 127.0.0.1:39434)
0620 02:40:47.570118 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:47.570183 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:47.570453 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:47.574224 (+ 3771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:47.574237 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:47.574291 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:47.575288 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.575311 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.577804 (+ 2493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.577809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.582641 (+ 4832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.582674 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.585891 (+ 3217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.585900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.586090 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.586123 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:40:47.586234 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 02:40:47.586417 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":1137,"thread_start_us":656,"threads_started":1}
I20260620 02:40:47.587715 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.571028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43945 (local address 127.0.0.1:52138)
0620 02:40:47.574384 (+ 3356us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:47.574408 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:47.574550 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:47.575756 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:47.575764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:47.575801 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:47.576454 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.576467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.578949 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.578957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.582460 (+ 3503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.582488 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.586465 (+ 3977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.586471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.586672 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.586702 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:40:47.586781 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 02:40:47.586951 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3192,"mutex_wait_us":11,"thread_start_us":3011,"threads_started":1}
I20260620 02:40:47.588116 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.569284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35705 (local address 127.0.0.1:38508)
0620 02:40:47.572498 (+ 3214us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:47.572516 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:47.572707 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:47.574418 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:47.574428 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:47.574475 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:47.575268 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.575286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.577669 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.577678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.583026 (+ 5348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.583035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.585363 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.585374 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.585734 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.585776 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 02:40:47.585958 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 02:40:47.586229 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3080,"mutex_wait_us":1472,"thread_start_us":2923,"threads_started":1}
I20260620 02:40:47.588744 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.571910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.0.97.3:43945)
0620 02:40:47.573818 (+ 1908us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:47.573831 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:47.574575 (+ 744us) server_negotiation.cc:407] Connection header received
0620 02:40:47.574721 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:47.574730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:47.575138 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:47.575395 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:47.576694 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.578441 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.582745 (+ 4304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.585120 (+ 2375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.585434 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.587976 (+ 2542us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:47.588108 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 02:40:47.588361 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1679,"thread_start_us":1503,"threads_started":1}
I20260620 02:40:47.589076 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.569345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.0.97.2:43195)
0620 02:40:47.573059 (+ 3714us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:47.573068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:47.573093 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:40:47.573198 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:47.573205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:47.573796 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:47.574776 (+ 980us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:47.575627 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.577541 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.582963 (+ 5422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.585120 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.585434 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.585635 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:47.588097 (+ 2462us) server_negotiation.cc:299] Negotiation successful
0620 02:40:47.588354 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3550,"thread_start_us":3429,"threads_started":1}
I20260620 02:40:47.593214 19095 tablet_service.cc:1467] Processing CreateTablet for tablet 900cc0c279f94f89b2f2db93809acd2c (DEFAULT_TABLE table=test-table [id=23b29ce7ece247849f07f8d2e1c0df68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:47.593214 18827 tablet_service.cc:1467] Processing CreateTablet for tablet 900cc0c279f94f89b2f2db93809acd2c (DEFAULT_TABLE table=test-table [id=23b29ce7ece247849f07f8d2e1c0df68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:47.593618 18961 tablet_service.cc:1467] Processing CreateTablet for tablet 900cc0c279f94f89b2f2db93809acd2c (DEFAULT_TABLE table=test-table [id=23b29ce7ece247849f07f8d2e1c0df68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:47.595242 18961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900cc0c279f94f89b2f2db93809acd2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:47.595368 18827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900cc0c279f94f89b2f2db93809acd2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:47.595368 19095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900cc0c279f94f89b2f2db93809acd2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:47.612205 19179 tablet_bootstrap.cc:492] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: Bootstrap starting.
I20260620 02:40:47.617064 19179 tablet_bootstrap.cc:654] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:47.618747 19179 log.cc:824] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:47.619537 19180 tablet_bootstrap.cc:492] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: Bootstrap starting.
I20260620 02:40:47.621876 19181 tablet_bootstrap.cc:492] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: Bootstrap starting.
I20260620 02:40:47.626585 19179 tablet_bootstrap.cc:492] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: No bootstrap required, opened a new log
I20260620 02:40:47.627573 19179 ts_tablet_manager.cc:1397] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260620 02:40:47.628396 19180 tablet_bootstrap.cc:654] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:47.628393 19181 tablet_bootstrap.cc:654] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:47.630092 19180 log.cc:824] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:47.630105 19181 log.cc:824] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:47.634837 19181 tablet_bootstrap.cc:492] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: No bootstrap required, opened a new log
I20260620 02:40:47.635239 19181 ts_tablet_manager.cc:1397] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260620 02:40:47.636484 19180 tablet_bootstrap.cc:492] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: No bootstrap required, opened a new log
I20260620 02:40:47.636803 19180 ts_tablet_manager.cc:1397] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.003s
I20260620 02:40:47.643134 19179 raft_consensus.cc:348] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a8d563291943929335bf59ad0cc51b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43195 } } peers { permanent_uuid: "5a6d1ca72d444115bec700e60beaf4f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35705 } } peers { permanent_uuid: "e94c39155e75406288b91fc7299cc85c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260620 02:40:47.644043 19179 raft_consensus.cc:374] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:47.644207 19179 raft_consensus.cc:729] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a8d563291943929335bf59ad0cc51b, State: Initialized, Role: FOLLOWER
I20260620 02:40:47.645699 19179 consensus_queue.cc:260] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a8d563291943929335bf59ad0cc51b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43195 } } peers { permanent_uuid: "5a6d1ca72d444115bec700e60beaf4f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35705 } } peers { permanent_uuid: "e94c39155e75406288b91fc7299cc85c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260620 02:40:47.647662 19026 heartbeater.cc:503] Master 127.0.97.62:39471 was elected leader, sending a full tablet report...
I20260620 02:40:47.648581 19179 ts_tablet_manager.cc:1428] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
I20260620 02:40:47.649096 19180 raft_consensus.cc:348] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a8d563291943929335bf59ad0cc51b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43195 } } peers { permanent_uuid: "5a6d1ca72d444115bec700e60beaf4f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35705 } } peers { permanent_uuid: "e94c39155e75406288b91fc7299cc85c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260620 02:40:47.649721 19180 raft_consensus.cc:374] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:47.649863 19180 raft_consensus.cc:729] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6d1ca72d444115bec700e60beaf4f1, State: Initialized, Role: FOLLOWER
I20260620 02:40:47.649708 19181 raft_consensus.cc:348] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a8d563291943929335bf59ad0cc51b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43195 } } peers { permanent_uuid: "5a6d1ca72d444115bec700e60beaf4f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35705 } } peers { permanent_uuid: "e94c39155e75406288b91fc7299cc85c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260620 02:40:47.650185 19181 raft_consensus.cc:374] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:47.650283 19181 raft_consensus.cc:729] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94c39155e75406288b91fc7299cc85c, State: Initialized, Role: FOLLOWER
I20260620 02:40:47.650679 19180 consensus_queue.cc:260] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a8d563291943929335bf59ad0cc51b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43195 } } peers { permanent_uuid: "5a6d1ca72d444115bec700e60beaf4f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35705 } } peers { permanent_uuid: "e94c39155e75406288b91fc7299cc85c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260620 02:40:47.650871 19181 consensus_queue.cc:260] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a8d563291943929335bf59ad0cc51b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43195 } } peers { permanent_uuid: "5a6d1ca72d444115bec700e60beaf4f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35705 } } peers { permanent_uuid: "e94c39155e75406288b91fc7299cc85c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260620 02:40:47.652760 19180 ts_tablet_manager.cc:1428] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260620 02:40:47.654472 19160 heartbeater.cc:503] Master 127.0.97.62:39471 was elected leader, sending a full tablet report...
I20260620 02:40:47.655409 19181 ts_tablet_manager.cc:1428] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: Time spent starting tablet: real 0.020s user 0.009s sys 0.014s
I20260620 02:40:47.664187 19187 raft_consensus.cc:482] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:40:47.664546 19187 raft_consensus.cc:504] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a8d563291943929335bf59ad0cc51b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43195 } } peers { permanent_uuid: "5a6d1ca72d444115bec700e60beaf4f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35705 } } peers { permanent_uuid: "e94c39155e75406288b91fc7299cc85c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260620 02:40:47.666340 19187 leader_election.cc:288] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a8d563291943929335bf59ad0cc51b (127.0.97.2:43195), 5a6d1ca72d444115bec700e60beaf4f1 (127.0.97.1:35705)
I20260620 02:40:47.674256 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.666748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43195 (local address 127.0.97.3:41775)
0620 02:40:47.666984 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:47.667008 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:47.667189 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:47.668004 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:47.668007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:47.668037 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:47.668273 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.668281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.669360 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.669366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.670745 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.670759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.672787 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.672794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.673299 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.673360 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 02:40:47.673537 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 02:40:47.673716 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 02:40:47.674721 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.666800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41775 (local address 127.0.97.2:43195)
0620 02:40:47.667010 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:47.667017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:47.667456 (+ 439us) server_negotiation.cc:407] Connection header received
0620 02:40:47.667544 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:47.667548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:47.667648 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:47.667763 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:47.668461 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.669170 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.670937 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.671780 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.672750 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.672860 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:47.673663 (+ 803us) server_negotiation.cc:299] Negotiation successful
0620 02:40:47.674151 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:40:47.675218 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.667595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39701 (local address 127.0.97.1:35705)
0620 02:40:47.668157 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:47.668165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:47.668349 (+ 184us) server_negotiation.cc:407] Connection header received
0620 02:40:47.668548 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:47.668554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:47.668629 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:47.668733 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:47.669380 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.670122 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.672610 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.673971 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.674219 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.674352 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:47.674802 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 02:40:47.674963 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 02:40:47.675257 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.666861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35705 (local address 127.0.97.3:39701)
0620 02:40:47.668222 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:47.668243 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:47.668394 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:47.668884 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:47.668889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:47.668919 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:47.669207 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.669216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.670288 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.670293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.671769 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:47.671783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.674310 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.674317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:47.674606 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.674636 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:40:47.674767 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 02:40:47.674914 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"thread_start_us":462,"threads_started":1}
I20260620 02:40:47.675822 18981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900cc0c279f94f89b2f2db93809acd2c" candidate_uuid: "e94c39155e75406288b91fc7299cc85c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a8d563291943929335bf59ad0cc51b" is_pre_election: true
I20260620 02:40:47.676481 18981 raft_consensus.cc:2454] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94c39155e75406288b91fc7299cc85c in term 0.
I20260620 02:40:47.676533 18847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900cc0c279f94f89b2f2db93809acd2c" candidate_uuid: "e94c39155e75406288b91fc7299cc85c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6d1ca72d444115bec700e60beaf4f1" is_pre_election: true
I20260620 02:40:47.677109 18847 raft_consensus.cc:2454] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94c39155e75406288b91fc7299cc85c in term 0.
I20260620 02:40:47.677306 19051 leader_election.cc:302] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6a8d563291943929335bf59ad0cc51b, e94c39155e75406288b91fc7299cc85c; no voters:
I20260620 02:40:47.677762 19187 raft_consensus.cc:2781] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:40:47.677937 19187 raft_consensus.cc:482] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:40:47.678062 19187 raft_consensus.cc:3037] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:47.680728 19187 raft_consensus.cc:504] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a8d563291943929335bf59ad0cc51b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43195 } } peers { permanent_uuid: "5a6d1ca72d444115bec700e60beaf4f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35705 } } peers { permanent_uuid: "e94c39155e75406288b91fc7299cc85c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260620 02:40:47.682107 19187 leader_election.cc:288] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [CANDIDATE]: Term 1 election: Requested vote from peers d6a8d563291943929335bf59ad0cc51b (127.0.97.2:43195), 5a6d1ca72d444115bec700e60beaf4f1 (127.0.97.1:35705)
I20260620 02:40:47.683029 18847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900cc0c279f94f89b2f2db93809acd2c" candidate_uuid: "e94c39155e75406288b91fc7299cc85c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6d1ca72d444115bec700e60beaf4f1"
I20260620 02:40:47.683368 18847 raft_consensus.cc:3037] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:47.683346 18981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900cc0c279f94f89b2f2db93809acd2c" candidate_uuid: "e94c39155e75406288b91fc7299cc85c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a8d563291943929335bf59ad0cc51b"
I20260620 02:40:47.683670 18981 raft_consensus.cc:3037] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:47.686765 18847 raft_consensus.cc:2454] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94c39155e75406288b91fc7299cc85c in term 1.
I20260620 02:40:47.687523 19049 leader_election.cc:302] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a6d1ca72d444115bec700e60beaf4f1, e94c39155e75406288b91fc7299cc85c; no voters:
I20260620 02:40:47.687971 19187 raft_consensus.cc:2781] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:40:47.687942 18981 raft_consensus.cc:2454] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94c39155e75406288b91fc7299cc85c in term 1.
I20260620 02:40:47.688597 19187 raft_consensus.cc:686] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 1 LEADER]: Becoming Leader. State: Replica: e94c39155e75406288b91fc7299cc85c, State: Running, Role: LEADER
W20260620 02:40:47.689517 19161 tablet.cc:2369] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:40:47.689399 19187 consensus_queue.cc:237] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a8d563291943929335bf59ad0cc51b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43195 } } peers { permanent_uuid: "5a6d1ca72d444115bec700e60beaf4f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35705 } } peers { permanent_uuid: "e94c39155e75406288b91fc7299cc85c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260620 02:40:47.695412 18703 catalog_manager.cc:5495] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c reported cstate change: term changed from 0 to 1, leader changed from <none> to e94c39155e75406288b91fc7299cc85c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e94c39155e75406288b91fc7299cc85c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a8d563291943929335bf59ad0cc51b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a6d1ca72d444115bec700e60beaf4f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e94c39155e75406288b91fc7299cc85c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } health_report { overall_health: HEALTHY } } }
W20260620 02:40:47.702379 19027 tablet.cc:2369] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:40:47.716540 18893 tablet.cc:2369] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:40:47.727818 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.721187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.0.97.3:43945)
0620 02:40:47.721456 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:47.721463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:47.721558 (+ 95us) server_negotiation.cc:407] Connection header received
0620 02:40:47.722045 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:47.722050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:47.722138 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:47.722319 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:40:47.722778 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.723569 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.725121 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.725551 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.725702 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.726469 (+ 767us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:40:47.727140 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:40:47.727352 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:47.727491 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 02:40:47.727543 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 02:40:47.740725 18981 raft_consensus.cc:1261] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b [term 1 FOLLOWER]: Refusing update from remote peer e94c39155e75406288b91fc7299cc85c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:40:47.741163 18847 raft_consensus.cc:1261] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1 [term 1 FOLLOWER]: Refusing update from remote peer e94c39155e75406288b91fc7299cc85c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:40:47.742947 19189 consensus_queue.cc:1035] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a8d563291943929335bf59ad0cc51b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:40:47.743417 19187 consensus_queue.cc:1035] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a6d1ca72d444115bec700e60beaf4f1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:40:47.769599 19193 mvcc.cc:205] Tried to move back new op lower bound from 7298757622727057408 to 7298757622540365824. Current Snapshot: MvccSnapshot[applied={T|T < 7298757622727057408}]
I20260620 02:40:47.796826 18703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:40:47.797309 18703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:40:47.802783 18703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3bcbf4350a04766af2bdf3694778193: Sending DeleteTablet for 3 replicas of tablet 900cc0c279f94f89b2f2db93809acd2c
I20260620 02:40:47.804950 19095 tablet_service.cc:1514] Processing DeleteTablet for tablet 900cc0c279f94f89b2f2db93809acd2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52138
I20260620 02:40:47.804950 18827 tablet_service.cc:1514] Processing DeleteTablet for tablet 900cc0c279f94f89b2f2db93809acd2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38508
I20260620 02:40:47.805998 19202 tablet_replica.cc:330] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: stopping tablet replica
I20260620 02:40:47.806130 18961 tablet_service.cc:1514] Processing DeleteTablet for tablet 900cc0c279f94f89b2f2db93809acd2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39434
I20260620 02:40:47.806807 19203 tablet_replica.cc:330] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: stopping tablet replica
I20260620 02:40:47.807078 19202 raft_consensus.cc:2229] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:40:47.807240 19203 raft_consensus.cc:2229] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:40:47.807632 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 18764
I20260620 02:40:47.807884 19202 raft_consensus.cc:2258] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:47.808064 19203 raft_consensus.cc:2258] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:47.810595 19202 ts_tablet_manager.cc:1905] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:47.817451 19203 ts_tablet_manager.cc:1905] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:47.817584 19204 tablet_replica.cc:330] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: stopping tablet replica
I20260620 02:40:47.820417 19204 raft_consensus.cc:2229] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:40:47.821535 19202 ts_tablet_manager.cc:1918] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:47.821785 19202 log.cc:1196] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/900cc0c279f94f89b2f2db93809acd2c
I20260620 02:40:47.822712 19202 ts_tablet_manager.cc:1939] T 900cc0c279f94f89b2f2db93809acd2c P 5a6d1ca72d444115bec700e60beaf4f1: Deleting consensus metadata
I20260620 02:40:47.825913 18688 catalog_manager.cc:4841] TS 5a6d1ca72d444115bec700e60beaf4f1 (127.0.97.1:35705): tablet 900cc0c279f94f89b2f2db93809acd2c (table test-table [id=23b29ce7ece247849f07f8d2e1c0df68]) successfully deleted
I20260620 02:40:47.826764 19204 raft_consensus.cc:2258] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:47.828727 19203 ts_tablet_manager.cc:1918] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:47.828962 19203 log.cc:1196] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/900cc0c279f94f89b2f2db93809acd2c
I20260620 02:40:47.829617 19203 ts_tablet_manager.cc:1939] T 900cc0c279f94f89b2f2db93809acd2c P e94c39155e75406288b91fc7299cc85c: Deleting consensus metadata
I20260620 02:40:47.830833 19204 ts_tablet_manager.cc:1905] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:47.831964 18690 catalog_manager.cc:4841] TS e94c39155e75406288b91fc7299cc85c (127.0.97.3:43945): tablet 900cc0c279f94f89b2f2db93809acd2c (table test-table [id=23b29ce7ece247849f07f8d2e1c0df68]) successfully deleted
Jun 20 02:40:47 dist-test-slave-rpg6 krb5kdc[18658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:40:47.840579 19204 ts_tablet_manager.cc:1918] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:47.840848 19204 log.cc:1196] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/900cc0c279f94f89b2f2db93809acd2c
I20260620 02:40:47.841727 19204 ts_tablet_manager.cc:1939] T 900cc0c279f94f89b2f2db93809acd2c P d6a8d563291943929335bf59ad0cc51b: Deleting consensus metadata
I20260620 02:40:47.844123 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:47.809660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.0.97.1:35705)
0620 02:40:47.809857 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:47.809864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:47.814824 (+ 4960us) server_negotiation.cc:407] Connection header received
0620 02:40:47.815222 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:47.815234 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:47.815373 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:47.815508 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:47.821121 (+ 5613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.823303 (+ 2182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.827518 (+ 4215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:47.828243 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:47.828400 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:47.834727 (+ 6327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:47.834777 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:47.834793 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:47.834851 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:47.838806 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:47.839714 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:47.839722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:47.839725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:47.839958 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:47.840414 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:47.841021 (+ 607us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:47.841028 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:47.842157 (+ 1129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:47.842539 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:47.843185 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 02:40:47.843653 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:40:47.844272 18690 catalog_manager.cc:4841] TS d6a8d563291943929335bf59ad0cc51b (127.0.97.2:43195): tablet 900cc0c279f94f89b2f2db93809acd2c (table test-table [id=23b29ce7ece247849f07f8d2e1c0df68]) successfully deleted
I20260620 02:40:47.844961 18887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:47.979065 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 18764
I20260620 02:40:48.011035 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 18898
Jun 20 02:40:48 dist-test-slave-rpg6 krb5kdc[18658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:40:48.047492 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:48.012142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.0.97.2:43195)
0620 02:40:48.012421 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:48.012442 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:48.012467 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:40:48.012542 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:48.012546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:48.012630 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:48.012751 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:48.015543 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:48.017087 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:48.019562 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:48.020486 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:48.020686 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:48.029322 (+ 8636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:48.029342 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:48.029360 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:48.029411 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:48.036576 (+ 7165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:48.041090 (+ 4514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:48.041100 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:48.041103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:48.041926 (+ 823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:48.043137 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:48.043142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:48.043145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:48.044054 (+ 909us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:48.044704 (+ 650us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:48.046266 (+ 1562us) server_negotiation.cc:299] Negotiation successful
0620 02:40:48.047108 (+ 842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:40:48.049625 19021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:48.176626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 18898
I20260620 02:40:48.200553 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 19032
Jun 20 02:40:48 dist-test-slave-rpg6 krb5kdc[18658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:40:48.228814 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:48.201702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.0.97.3:43945)
0620 02:40:48.201937 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:48.201943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:48.202103 (+ 160us) server_negotiation.cc:407] Connection header received
0620 02:40:48.202400 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:48.202407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:48.202540 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:48.202700 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:48.205057 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:48.206292 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:48.208420 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:48.209170 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:48.209413 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:48.217152 (+ 7739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:48.217181 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:48.217211 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:48.217279 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:48.223237 (+ 5958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:48.225192 (+ 1955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:48.225206 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:48.225210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:48.225448 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:48.226581 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:48.226588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:48.226591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:48.227171 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:48.227564 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:48.227956 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 02:40:48.228272 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 02:40:48.231318 19155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:48.441859 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 19032
I20260620 02:40:48.469204 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 18671
I20260620 02:40:48.470854 18733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:48.602963 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 18671
2026-06-20T02:40:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5726 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T02:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:40:48Z Disabled control of system clock
I20260620 02:40:48.715574 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41541
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:40:49.182145 19213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:49.182951 19213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:49.183214 19213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:49.193187 19213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:40:49.193434 19213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:49.193513 19213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:49.193571 19213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:40:49.193632 19213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:40:49.206091 19213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:49.207655 19213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:49.211570 19213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:49.229535 19218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:49.231650 19219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:49.232084 19221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:49.234170 19213 server_base.cc:1034] running on GCE node
I20260620 02:40:49.236613 19213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:49.238986 19213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:49.240428 19213 hybrid_clock.cc:648] HybridClock initialized: now 1781923249240263 us; error 123 us; skew 500 ppm
I20260620 02:40:49.241415 19213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:40:49.245275 19213 webserver.cc:460] Webserver started at http://127.0.97.62:42189/ using document root <none> and password file <none>
I20260620 02:40:49.246179 19213 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:49.246343 19213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:49.246730 19213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:49.250192 19213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "6c91ecc342404860917d19ca467b7f3a"
format_stamp: "Formatted at 2026-06-20 02:40:49 on dist-test-slave-rpg6"
server_key: "7b8ac243d0cd80b899ebec8b92053557"
server_key_iv: "05b04f6fd14df14caeb1febc83de6f3c"
server_key_version: "encryptionkey@0"
I20260620 02:40:49.251354 19213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "6c91ecc342404860917d19ca467b7f3a"
format_stamp: "Formatted at 2026-06-20 02:40:49 on dist-test-slave-rpg6"
server_key: "7b8ac243d0cd80b899ebec8b92053557"
server_key_iv: "05b04f6fd14df14caeb1febc83de6f3c"
server_key_version: "encryptionkey@0"
I20260620 02:40:49.258639 19213 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260620 02:40:49.263347 19227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:49.266106 19213 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:40:49.266428 19213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "6c91ecc342404860917d19ca467b7f3a"
format_stamp: "Formatted at 2026-06-20 02:40:49 on dist-test-slave-rpg6"
server_key: "7b8ac243d0cd80b899ebec8b92053557"
server_key_iv: "05b04f6fd14df14caeb1febc83de6f3c"
server_key_version: "encryptionkey@0"
I20260620 02:40:49.266754 19213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:49.298683 19213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:49.299908 19213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:49.300225 19213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:49.328586 19278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41541 every 8 connection(s)
I20260620 02:40:49.328678 19213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41541
I20260620 02:40:49.330673 19213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:40:49.337697 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 19213
I20260620 02:40:49.337975 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:40:49.338193 19279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:49.338707 388 external_mini_cluster.cc:1428] Setting key 51a0e869fae7aa92b3c1c6a1b82f1f7d
I20260620 02:40:49.361802 19282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:49.342995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.0.97.62:41541)
0620 02:40:49.344497 (+ 1502us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:49.344512 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:49.344575 (+ 63us) server_negotiation.cc:407] Connection header received
0620 02:40:49.347122 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:49.347155 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:49.347806 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:49.348356 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:49.349400 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:49.350982 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:49.354523 (+ 3541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:49.355649 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:49.356102 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:49.357830 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:49.357851 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:49.357878 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:49.358967 (+ 1089us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:49.359115 (+ 148us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:49.359292 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:49.359460 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:49.359877 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 02:40:49.360334 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":577,"threads_started":1}
I20260620 02:40:49.361814 19279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a: Bootstrap starting.
I20260620 02:40:49.368034 19279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:49.369633 19279 log.cc:824] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:49.375203 19279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a: No bootstrap required, opened a new log
I20260620 02:40:49.387537 19279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c91ecc342404860917d19ca467b7f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } }
I20260620 02:40:49.388067 19279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:49.388152 19279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c91ecc342404860917d19ca467b7f3a, State: Initialized, Role: FOLLOWER
I20260620 02:40:49.388875 19279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c91ecc342404860917d19ca467b7f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } }
I20260620 02:40:49.389127 19279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:40:49.389241 19279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:40:49.389395 19279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:49.392364 19279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c91ecc342404860917d19ca467b7f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } }
I20260620 02:40:49.392993 19279 leader_election.cc:302] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c91ecc342404860917d19ca467b7f3a; no voters:
I20260620 02:40:49.393731 19279 leader_election.cc:288] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:40:49.393929 19284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:40:49.394856 19284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [term 1 LEADER]: Becoming Leader. State: Replica: 6c91ecc342404860917d19ca467b7f3a, State: Running, Role: LEADER
I20260620 02:40:49.395596 19284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c91ecc342404860917d19ca467b7f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } }
I20260620 02:40:49.396556 19279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:40:49.398608 19286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c91ecc342404860917d19ca467b7f3a. Latest consensus state: current_term: 1 leader_uuid: "6c91ecc342404860917d19ca467b7f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c91ecc342404860917d19ca467b7f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } } }
I20260620 02:40:49.399058 19286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:49.403612 19290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:40:49.403364 19285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c91ecc342404860917d19ca467b7f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c91ecc342404860917d19ca467b7f3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41541 } } }
I20260620 02:40:49.403790 19285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:49.414346 19290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:40:49.429788 19290 catalog_manager.cc:1347] Generated new cluster ID: bed8f0cfe98b43e798756f119896de8f
I20260620 02:40:49.429966 19290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:40:49.446003 19290 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:40:49.448345 19290 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:40:49.464659 19290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a: Generated new TSK 0
I20260620 02:40:49.466428 19290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:40:49.479255 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--builtin_ntp_servers=127.0.97.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:40:49.981802 19304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:49.982156 19304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:49.982445 19304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:49.991690 19304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:49.991891 19304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:49.992053 19304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:40:50.005247 19304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:50.006834 19304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:50.009477 19304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:50.033692 19309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:50.034504 19310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:50.034915 19312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:50.036743 19304 server_base.cc:1034] running on GCE node
I20260620 02:40:50.037843 19304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:50.039463 19304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:50.040884 19304 hybrid_clock.cc:648] HybridClock initialized: now 1781923250040815 us; error 103 us; skew 500 ppm
I20260620 02:40:50.041344 19304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:40:50.052589 19304 webserver.cc:460] Webserver started at http://127.0.97.1:42175/ using document root <none> and password file <none>
I20260620 02:40:50.053501 19304 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:50.053682 19304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:50.054418 19304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:50.059557 19304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "c52432bf062049b680b8c9f573c1aa07"
format_stamp: "Formatted at 2026-06-20 02:40:50 on dist-test-slave-rpg6"
server_key: "154dbb496bb1ee47c916016ee9c145dc"
server_key_iv: "68acb3120f680c9a6ccaa090ae978a9a"
server_key_version: "encryptionkey@0"
I20260620 02:40:50.061681 19304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c52432bf062049b680b8c9f573c1aa07"
format_stamp: "Formatted at 2026-06-20 02:40:50 on dist-test-slave-rpg6"
server_key: "154dbb496bb1ee47c916016ee9c145dc"
server_key_iv: "68acb3120f680c9a6ccaa090ae978a9a"
server_key_version: "encryptionkey@0"
I20260620 02:40:50.073750 19304 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.007s sys 0.004s
I20260620 02:40:50.080950 19318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:50.084298 19304 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:40:50.084514 19304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "c52432bf062049b680b8c9f573c1aa07"
format_stamp: "Formatted at 2026-06-20 02:40:50 on dist-test-slave-rpg6"
server_key: "154dbb496bb1ee47c916016ee9c145dc"
server_key_iv: "68acb3120f680c9a6ccaa090ae978a9a"
server_key_version: "encryptionkey@0"
I20260620 02:40:50.084823 19304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:50.122207 19304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:50.123484 19304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:50.124552 19304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:50.125926 19304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:50.129257 19304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:50.129396 19304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:50.129583 19304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:50.129645 19304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:50.187783 19304 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34007
I20260620 02:40:50.187873 19430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34007 every 8 connection(s)
I20260620 02:40:50.189369 19304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:40:50.195519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 19304
I20260620 02:40:50.195714 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:40:50.196269 388 external_mini_cluster.cc:1428] Setting key 3f679163419bc46de33c2b44c3eb6ff6
I20260620 02:40:50.200706 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--builtin_ntp_servers=127.0.97.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:40:50.213629 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:50.193195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53505 (local address 127.0.97.62:41541)
0620 02:40:50.193780 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:50.193787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:50.194991 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 02:40:50.198490 (+ 3499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:50.198499 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:50.198765 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:50.198952 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:50.201743 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:50.203013 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:50.206329 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:50.207213 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:50.208397 (+ 1184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:50.212461 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:50.212467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:50.212471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:50.212496 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:50.212579 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:50.212596 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:50.212680 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:50.213087 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 02:40:50.213224 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":308,"threads_started":1}
I20260620 02:40:50.215205 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:50.193512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41541 (local address 127.0.97.1:53505)
0620 02:40:50.194754 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:50.194816 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:50.197926 (+ 3110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:50.199649 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:50.199668 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:50.200030 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:50.201305 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:50.201344 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:50.203271 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:50.203280 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:50.205814 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:50.205848 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:50.208843 (+ 2995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:50.208861 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:50.209419 (+ 558us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:50.211353 (+ 1934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:40:50.211362 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:50.211468 (+ 106us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:40:50.211667 (+ 199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:50.212731 (+ 1064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:50.212750 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:50.212761 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:40:50.213054 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 02:40:50.213621 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":409,"threads_started":1}
I20260620 02:40:50.218061 19431 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41541
I20260620 02:40:50.218572 19431 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:50.219976 19431 heartbeater.cc:511] Master 127.0.97.62:41541 requested a full tablet report, sending...
I20260620 02:40:50.223115 19244 ts_manager.cc:194] Registered new tserver with Master: c52432bf062049b680b8c9f573c1aa07 (127.0.97.1:34007)
I20260620 02:40:50.225471 19244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:53505
W20260620 02:40:50.678375 19435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:50.678925 19435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:50.679116 19435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:50.690379 19435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:50.690553 19435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:50.690745 19435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:40:50.702088 19435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:50.703585 19435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:50.705853 19435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:50.724718 19440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:50.726702 19443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:50.726775 19441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:50.728318 19435 server_base.cc:1034] running on GCE node
I20260620 02:40:50.731009 19435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:50.732843 19435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:50.734227 19435 hybrid_clock.cc:648] HybridClock initialized: now 1781923250734158 us; error 79 us; skew 500 ppm
I20260620 02:40:50.734668 19435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:40:50.739805 19435 webserver.cc:460] Webserver started at http://127.0.97.2:41065/ using document root <none> and password file <none>
I20260620 02:40:50.742063 19435 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:50.742395 19435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:50.743247 19435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:50.749217 19435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "16cbd1ecc23943a395bb11d12a1b45f5"
format_stamp: "Formatted at 2026-06-20 02:40:50 on dist-test-slave-rpg6"
server_key: "8e40ba0d49c0fe6fd6f91941b4c00b11"
server_key_iv: "9420a436652ee87d7bc206e5c4ce1afa"
server_key_version: "encryptionkey@0"
I20260620 02:40:50.751044 19435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "16cbd1ecc23943a395bb11d12a1b45f5"
format_stamp: "Formatted at 2026-06-20 02:40:50 on dist-test-slave-rpg6"
server_key: "8e40ba0d49c0fe6fd6f91941b4c00b11"
server_key_iv: "9420a436652ee87d7bc206e5c4ce1afa"
server_key_version: "encryptionkey@0"
I20260620 02:40:50.758370 19435 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260620 02:40:50.765230 19449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:50.767406 19435 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.005s
I20260620 02:40:50.767719 19435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "16cbd1ecc23943a395bb11d12a1b45f5"
format_stamp: "Formatted at 2026-06-20 02:40:50 on dist-test-slave-rpg6"
server_key: "8e40ba0d49c0fe6fd6f91941b4c00b11"
server_key_iv: "9420a436652ee87d7bc206e5c4ce1afa"
server_key_version: "encryptionkey@0"
I20260620 02:40:50.768193 19435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:50.794922 19435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:50.796007 19435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:50.796322 19435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:50.797523 19435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:50.799492 19435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:50.799602 19435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:50.799685 19435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:50.799733 19435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:50.903309 19561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37327 every 8 connection(s)
I20260620 02:40:50.903258 19435 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37327
I20260620 02:40:50.905871 19435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:40:50.907208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 19435
I20260620 02:40:50.908126 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:40:50.908798 388 external_mini_cluster.cc:1428] Setting key a46a902763ead445fcd3336b9eea213b
I20260620 02:40:50.912372 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--builtin_ntp_servers=127.0.97.20:40483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:40:50.934865 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:50.912660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53723 (local address 127.0.97.62:41541)
0620 02:40:50.913261 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:50.913270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:50.916200 (+ 2930us) server_negotiation.cc:407] Connection header received
0620 02:40:50.919809 (+ 3609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:50.919817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:50.920225 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:50.920469 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:50.922967 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:50.924401 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:50.930408 (+ 6007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:50.930994 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:50.931207 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:50.933800 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:50.933808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:50.933813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:50.933851 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:50.933907 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:50.933924 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:50.934051 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:50.934350 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 02:40:50.934474 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":300,"threads_started":1}
I20260620 02:40:50.937364 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:50.913544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41541 (local address 127.0.97.2:53723)
0620 02:40:50.915771 (+ 2227us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:50.915870 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:50.919258 (+ 3388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:50.920736 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:50.920754 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:50.920927 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:50.922448 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:50.922489 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:50.925558 (+ 3069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:50.925568 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:50.929703 (+ 4135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:50.929739 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:50.931364 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:50.931376 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:50.931663 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:50.932983 (+ 1320us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:40:50.932986 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:50.933239 (+ 253us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:40:50.933397 (+ 158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:50.934056 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:50.934072 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:50.934085 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:40:50.934318 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 02:40:50.934872 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"thread_start_us":416,"threads_started":1}
I20260620 02:40:50.940456 19562 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41541
I20260620 02:40:50.941071 19562 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:50.943686 19562 heartbeater.cc:511] Master 127.0.97.62:41541 requested a full tablet report, sending...
I20260620 02:40:50.947692 19244 ts_manager.cc:194] Registered new tserver with Master: 16cbd1ecc23943a395bb11d12a1b45f5 (127.0.97.2:37327)
I20260620 02:40:50.948637 19244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:53723
I20260620 02:40:51.230734 19431 heartbeater.cc:503] Master 127.0.97.62:41541 was elected leader, sending a full tablet report...
W20260620 02:40:51.450150 19566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:51.450650 19566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:51.450774 19566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:51.460561 19566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:51.460696 19566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:51.460829 19566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:40:51.472160 19566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:51.474445 19566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:51.477340 19566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:51.498545 19572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:51.498684 19574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:51.498545 19571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:51.500124 19566 server_base.cc:1034] running on GCE node
I20260620 02:40:51.500896 19566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:51.502175 19566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:51.503808 19566 hybrid_clock.cc:648] HybridClock initialized: now 1781923251503707 us; error 86 us; skew 500 ppm
I20260620 02:40:51.504611 19566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:40:51.508411 19566 webserver.cc:460] Webserver started at http://127.0.97.3:44971/ using document root <none> and password file <none>
I20260620 02:40:51.510041 19566 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:51.510250 19566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:51.510718 19566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:51.516418 19566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "c34d28852af44e128660232174ada1fa"
format_stamp: "Formatted at 2026-06-20 02:40:51 on dist-test-slave-rpg6"
server_key: "b1066c387e5e611cc28cd0297abeab6b"
server_key_iv: "583cabae4a34da59150ffa139d3da04e"
server_key_version: "encryptionkey@0"
I20260620 02:40:51.517994 19566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c34d28852af44e128660232174ada1fa"
format_stamp: "Formatted at 2026-06-20 02:40:51 on dist-test-slave-rpg6"
server_key: "b1066c387e5e611cc28cd0297abeab6b"
server_key_iv: "583cabae4a34da59150ffa139d3da04e"
server_key_version: "encryptionkey@0"
I20260620 02:40:51.529042 19566 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.004s
I20260620 02:40:51.535969 19580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:51.538396 19566 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:40:51.538688 19566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "c34d28852af44e128660232174ada1fa"
format_stamp: "Formatted at 2026-06-20 02:40:51 on dist-test-slave-rpg6"
server_key: "b1066c387e5e611cc28cd0297abeab6b"
server_key_iv: "583cabae4a34da59150ffa139d3da04e"
server_key_version: "encryptionkey@0"
I20260620 02:40:51.539281 19566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:51.591162 19566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:51.592486 19566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:51.592898 19566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:51.594166 19566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:51.596459 19566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:51.596609 19566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:51.596930 19566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:51.597210 19566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:51.659405 19566 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40911
I20260620 02:40:51.659462 19692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40911 every 8 connection(s)
I20260620 02:40:51.661603 19566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:40:51.669469 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 19566
I20260620 02:40:51.669711 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:40:51.670282 388 external_mini_cluster.cc:1428] Setting key 9b2c461254744b36e8a6fa0350948141
I20260620 02:40:51.680208 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.665377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33729 (local address 127.0.97.62:41541)
0620 02:40:51.665873 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:51.665879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:51.667342 (+ 1463us) server_negotiation.cc:407] Connection header received
0620 02:40:51.669556 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:51.669563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:51.669682 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:51.669798 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:51.671119 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.672320 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.675939 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.676462 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.676653 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.679066 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:51.679072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:51.679075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:51.679101 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:51.679172 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:51.679188 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:51.679306 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:51.679587 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 02:40:51.679733 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":269,"threads_started":1}
I20260620 02:40:51.682104 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.665922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41541 (local address 127.0.97.3:33729)
0620 02:40:51.667130 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:51.667172 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:51.669197 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:51.670072 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:51.670082 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:51.670191 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:51.670906 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.670926 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.672613 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.672617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.675594 (+ 2977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.675629 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.676823 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.676833 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.677501 (+ 668us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.678527 (+ 1026us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:40:51.678530 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:51.678590 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:40:51.678714 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:51.679313 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:51.679329 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:51.679339 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 02:40:51.679585 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 02:40:51.680220 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":359,"threads_started":1}
I20260620 02:40:51.684466 19693 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41541
I20260620 02:40:51.684814 19693 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:51.685984 19693 heartbeater.cc:511] Master 127.0.97.62:41541 requested a full tablet report, sending...
I20260620 02:40:51.688097 19244 ts_manager.cc:194] Registered new tserver with Master: c34d28852af44e128660232174ada1fa (127.0.97.3:40911)
I20260620 02:40:51.689265 19244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:33729
I20260620 02:40:51.689751 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:40:51.706919 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.697490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.0.97.62:41541)
0620 02:40:51.698086 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:51.698096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:51.698672 (+ 576us) server_negotiation.cc:407] Connection header received
0620 02:40:51.698965 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:51.698971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:51.699058 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:51.699184 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:51.699899 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.700770 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.702994 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.703447 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.703631 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.705788 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:51.705794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:51.705798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:51.705824 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:51.705976 (+ 152us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:51.706036 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:51.706248 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:51.706511 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 02:40:51.706580 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260620 02:40:51.713078 19244 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:40:51.723785 19244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:40:51.771809 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.747251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.0.97.1:34007)
0620 02:40:51.752828 (+ 5577us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:51.752839 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:51.752878 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:40:51.753005 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:51.753014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:51.754214 (+ 1200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:51.756588 (+ 2374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:51.757146 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.760357 (+ 3211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.767514 (+ 7157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.769844 (+ 2330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.770023 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.770368 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:51.771172 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 02:40:51.771366 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5306,"thread_start_us":5368,"threads_started":1}
I20260620 02:40:51.772114 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.747034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34007 (local address 127.0.0.1:55336)
0620 02:40:51.751433 (+ 4399us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:51.751500 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:51.751973 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:51.755263 (+ 3290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:51.755275 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:51.755334 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:51.756248 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.756265 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.760653 (+ 4388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.760665 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.766880 (+ 6215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.766918 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.770539 (+ 3621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.770547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.770798 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.770874 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 02:40:51.771053 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 02:40:51.771699 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1907,"mutex_wait_us":382,"thread_start_us":1497,"threads_started":1}
I20260620 02:40:51.776765 19366 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3aeb6ecc7f49d9a88d9c60858516d1 (DEFAULT_TABLE table=test-table [id=30b71add69e64bd193a71c6a023986bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:51.778311 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.747709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.0.97.3:40911)
0620 02:40:51.761006 (+ 13297us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:51.761021 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:51.761079 (+ 58us) server_negotiation.cc:407] Connection header received
0620 02:40:51.761505 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:51.761522 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:51.762286 (+ 764us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:51.763184 (+ 898us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:51.767219 (+ 4035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.769204 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.774084 (+ 4880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.775796 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.775971 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.776978 (+ 1007us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:51.777706 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 02:40:51.777917 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12998,"thread_start_us":16444,"threads_started":1}
I20260620 02:40:51.778338 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.753104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37327 (local address 127.0.0.1:49194)
0620 02:40:51.768103 (+ 14999us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:51.768132 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:51.768340 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:51.769442 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:51.769448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:51.769486 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:51.770407 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.770429 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.773314 (+ 2885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.773321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.775482 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.775503 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.777515 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.777522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.777695 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.777729 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:40:51.777804 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 02:40:51.777937 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14706,"mutex_wait_us":8,"thread_start_us":14356,"threads_started":1}
I20260620 02:40:51.778640 19366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a3aeb6ecc7f49d9a88d9c60858516d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:51.778657 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.750429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.0.97.2:37327)
0620 02:40:51.767325 (+ 16896us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:51.767338 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:51.768512 (+ 1174us) server_negotiation.cc:407] Connection header received
0620 02:40:51.768702 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:51.768713 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:51.769121 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:51.770514 (+ 1393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:51.770971 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.773082 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.775752 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.777315 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.777487 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.777813 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:51.777900 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 02:40:51.778082 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16636,"thread_start_us":16486,"threads_started":1}
I20260620 02:40:51.778918 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.747134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40911 (local address 127.0.0.1:45988)
0620 02:40:51.759921 (+ 12787us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:51.759954 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:51.760185 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:51.764959 (+ 4774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:51.764987 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:51.765076 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:51.766539 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.766573 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.769442 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.769449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.773794 (+ 4345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.773813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.777123 (+ 3310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.777130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.777296 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.777336 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:40:51.777425 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 02:40:51.778599 (+ 1174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12585,"mutex_wait_us":1,"thread_start_us":12409,"threads_started":1}
I20260620 02:40:51.787542 19497 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3aeb6ecc7f49d9a88d9c60858516d1 (DEFAULT_TABLE table=test-table [id=30b71add69e64bd193a71c6a023986bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:51.787719 19628 tablet_service.cc:1467] Processing CreateTablet for tablet 4a3aeb6ecc7f49d9a88d9c60858516d1 (DEFAULT_TABLE table=test-table [id=30b71add69e64bd193a71c6a023986bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:51.789497 19497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a3aeb6ecc7f49d9a88d9c60858516d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:51.789743 19628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a3aeb6ecc7f49d9a88d9c60858516d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:51.808830 19712 tablet_bootstrap.cc:492] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07: Bootstrap starting.
I20260620 02:40:51.814225 19713 tablet_bootstrap.cc:492] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: Bootstrap starting.
I20260620 02:40:51.816908 19712 tablet_bootstrap.cc:654] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:51.820612 19712 log.cc:824] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:51.821549 19714 tablet_bootstrap.cc:492] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: Bootstrap starting.
I20260620 02:40:51.823813 19713 tablet_bootstrap.cc:654] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:51.825938 19712 tablet_bootstrap.cc:492] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07: No bootstrap required, opened a new log
I20260620 02:40:51.826448 19712 ts_tablet_manager.cc:1397] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.008s
I20260620 02:40:51.827430 19713 log.cc:824] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:51.829878 19714 tablet_bootstrap.cc:654] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:51.831806 19714 log.cc:824] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:51.833010 19713 tablet_bootstrap.cc:492] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: No bootstrap required, opened a new log
I20260620 02:40:51.833451 19713 ts_tablet_manager.cc:1397] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20260620 02:40:51.837085 19714 tablet_bootstrap.cc:492] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: No bootstrap required, opened a new log
I20260620 02:40:51.837476 19714 ts_tablet_manager.cc:1397] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260620 02:40:51.839393 19712 raft_consensus.cc:348] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52432bf062049b680b8c9f573c1aa07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34007 } } peers { permanent_uuid: "c34d28852af44e128660232174ada1fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40911 } } peers { permanent_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } }
I20260620 02:40:51.840430 19712 raft_consensus.cc:374] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:51.841065 19712 raft_consensus.cc:729] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52432bf062049b680b8c9f573c1aa07, State: Initialized, Role: FOLLOWER
I20260620 02:40:51.841979 19712 consensus_queue.cc:260] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52432bf062049b680b8c9f573c1aa07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34007 } } peers { permanent_uuid: "c34d28852af44e128660232174ada1fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40911 } } peers { permanent_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } }
I20260620 02:40:51.846388 19712 ts_tablet_manager.cc:1428] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260620 02:40:51.847168 19713 raft_consensus.cc:348] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52432bf062049b680b8c9f573c1aa07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34007 } } peers { permanent_uuid: "c34d28852af44e128660232174ada1fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40911 } } peers { permanent_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } }
I20260620 02:40:51.847633 19713 raft_consensus.cc:374] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:51.847770 19713 raft_consensus.cc:729] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34d28852af44e128660232174ada1fa, State: Initialized, Role: FOLLOWER
I20260620 02:40:51.848567 19713 consensus_queue.cc:260] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52432bf062049b680b8c9f573c1aa07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34007 } } peers { permanent_uuid: "c34d28852af44e128660232174ada1fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40911 } } peers { permanent_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } }
I20260620 02:40:51.850309 19693 heartbeater.cc:503] Master 127.0.97.62:41541 was elected leader, sending a full tablet report...
I20260620 02:40:51.851393 19713 ts_tablet_manager.cc:1428] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260620 02:40:51.853348 19714 raft_consensus.cc:348] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52432bf062049b680b8c9f573c1aa07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34007 } } peers { permanent_uuid: "c34d28852af44e128660232174ada1fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40911 } } peers { permanent_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } }
I20260620 02:40:51.853924 19714 raft_consensus.cc:374] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:51.854035 19714 raft_consensus.cc:729] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16cbd1ecc23943a395bb11d12a1b45f5, State: Initialized, Role: FOLLOWER
I20260620 02:40:51.855104 19714 consensus_queue.cc:260] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52432bf062049b680b8c9f573c1aa07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34007 } } peers { permanent_uuid: "c34d28852af44e128660232174ada1fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40911 } } peers { permanent_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } }
I20260620 02:40:51.857149 19562 heartbeater.cc:503] Master 127.0.97.62:41541 was elected leader, sending a full tablet report...
I20260620 02:40:51.857825 19714 ts_tablet_manager.cc:1428] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: Time spent starting tablet: real 0.020s user 0.009s sys 0.011s
I20260620 02:40:51.902973 19718 raft_consensus.cc:482] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:40:51.903584 19718 raft_consensus.cc:504] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52432bf062049b680b8c9f573c1aa07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34007 } } peers { permanent_uuid: "c34d28852af44e128660232174ada1fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40911 } } peers { permanent_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } }
I20260620 02:40:51.905612 19718 leader_election.cc:288] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c34d28852af44e128660232174ada1fa (127.0.97.3:40911), 16cbd1ecc23943a395bb11d12a1b45f5 (127.0.97.2:37327)
W20260620 02:40:51.909886 19563 tablet.cc:2369] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:40:51.914541 19694 tablet.cc:2369] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:40:51.916483 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.906182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52745 (local address 127.0.97.3:40911)
0620 02:40:51.906425 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:51.906431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:51.907138 (+ 707us) server_negotiation.cc:407] Connection header received
0620 02:40:51.907505 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:51.907510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:51.907577 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:51.907632 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:51.908599 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.909856 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.913221 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.915142 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.915480 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.915620 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:51.916065 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 02:40:51.916217 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:40:51.916620 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.906088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40911 (local address 127.0.97.1:52745)
0620 02:40:51.906985 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:51.907013 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:51.907295 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:51.907942 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:51.907946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:51.907978 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:51.908437 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.908449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.910099 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.910106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.912901 (+ 2795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.912928 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.915554 (+ 2626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.915562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.915783 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.915811 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 02:40:51.915999 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 02:40:51.916170 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":97,"thread_start_us":886,"threads_started":1}
I20260620 02:40:51.918253 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.906163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60819 (local address 127.0.97.2:37327)
0620 02:40:51.906384 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:51.906392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:51.907734 (+ 1342us) server_negotiation.cc:407] Connection header received
0620 02:40:51.907940 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:51.907946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:51.908058 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:51.908171 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:51.908922 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.910009 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.915098 (+ 5089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.916632 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.916777 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.916878 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:51.917794 (+ 916us) server_negotiation.cc:299] Negotiation successful
0620 02:40:51.917944 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 02:40:51.918370 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.906716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37327 (local address 127.0.97.1:60819)
0620 02:40:51.907593 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:51.907612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:51.907763 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:51.908218 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:51.908223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:51.908252 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:51.908695 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.908706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.910181 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.910185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.912902 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:51.912928 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.917256 (+ 4328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.917266 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:51.917583 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.917613 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:40:51.917736 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 02:40:51.918026 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":142,"thread_start_us":559,"threads_started":1}
I20260620 02:40:51.918533 19648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3aeb6ecc7f49d9a88d9c60858516d1" candidate_uuid: "c52432bf062049b680b8c9f573c1aa07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34d28852af44e128660232174ada1fa" is_pre_election: true
I20260620 02:40:51.919304 19648 raft_consensus.cc:2454] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c52432bf062049b680b8c9f573c1aa07 in term 0.
I20260620 02:40:51.920377 19321 leader_election.cc:302] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c34d28852af44e128660232174ada1fa, c52432bf062049b680b8c9f573c1aa07; no voters:
I20260620 02:40:51.921481 19718 raft_consensus.cc:2781] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:40:51.920390 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3aeb6ecc7f49d9a88d9c60858516d1" candidate_uuid: "c52432bf062049b680b8c9f573c1aa07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" is_pre_election: true
I20260620 02:40:51.921663 19718 raft_consensus.cc:482] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:40:51.921782 19718 raft_consensus.cc:3037] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:51.922366 19517 raft_consensus.cc:2454] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c52432bf062049b680b8c9f573c1aa07 in term 0.
I20260620 02:40:51.925626 19718 raft_consensus.cc:504] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52432bf062049b680b8c9f573c1aa07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34007 } } peers { permanent_uuid: "c34d28852af44e128660232174ada1fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40911 } } peers { permanent_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } }
I20260620 02:40:51.926484 19718 leader_election.cc:288] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [CANDIDATE]: Term 1 election: Requested vote from peers c34d28852af44e128660232174ada1fa (127.0.97.3:40911), 16cbd1ecc23943a395bb11d12a1b45f5 (127.0.97.2:37327)
I20260620 02:40:51.927227 19648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3aeb6ecc7f49d9a88d9c60858516d1" candidate_uuid: "c52432bf062049b680b8c9f573c1aa07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34d28852af44e128660232174ada1fa"
I20260620 02:40:51.927347 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a3aeb6ecc7f49d9a88d9c60858516d1" candidate_uuid: "c52432bf062049b680b8c9f573c1aa07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16cbd1ecc23943a395bb11d12a1b45f5"
I20260620 02:40:51.927456 19648 raft_consensus.cc:3037] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:51.927693 19517 raft_consensus.cc:3037] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:51.930514 19648 raft_consensus.cc:2454] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c52432bf062049b680b8c9f573c1aa07 in term 1.
I20260620 02:40:51.930836 19517 raft_consensus.cc:2454] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c52432bf062049b680b8c9f573c1aa07 in term 1.
I20260620 02:40:51.931293 19321 leader_election.cc:302] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c34d28852af44e128660232174ada1fa, c52432bf062049b680b8c9f573c1aa07; no voters:
I20260620 02:40:51.931988 19718 raft_consensus.cc:2781] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:40:51.932785 19718 raft_consensus.cc:686] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 1 LEADER]: Becoming Leader. State: Replica: c52432bf062049b680b8c9f573c1aa07, State: Running, Role: LEADER
I20260620 02:40:51.933373 19718 consensus_queue.cc:237] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52432bf062049b680b8c9f573c1aa07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34007 } } peers { permanent_uuid: "c34d28852af44e128660232174ada1fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40911 } } peers { permanent_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } }
I20260620 02:40:51.939756 19243 catalog_manager.cc:5495] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 reported cstate change: term changed from 0 to 1, leader changed from <none> to c52432bf062049b680b8c9f573c1aa07 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c52432bf062049b680b8c9f573c1aa07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52432bf062049b680b8c9f573c1aa07" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c34d28852af44e128660232174ada1fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 } health_report { overall_health: UNKNOWN } } }
W20260620 02:40:51.944175 19432 tablet.cc:2369] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:40:51.981761 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:51.973641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.0.97.1:34007)
0620 02:40:51.974404 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:51.974410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:51.974436 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:40:51.974617 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:51.974623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:51.974863 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:51.975078 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:40:51.975609 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.976419 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.978848 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:51.979287 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:51.979447 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:51.980501 (+ 1054us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:40:51.981050 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:40:51.981286 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:51.981458 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 02:40:51.981511 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260620 02:40:51.993245 19648 raft_consensus.cc:1261] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa [term 1 FOLLOWER]: Refusing update from remote peer c52432bf062049b680b8c9f573c1aa07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:40:51.993299 19517 raft_consensus.cc:1261] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5 [term 1 FOLLOWER]: Refusing update from remote peer c52432bf062049b680b8c9f573c1aa07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:40:51.994493 19723 consensus_queue.cc:1035] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34d28852af44e128660232174ada1fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:40:51.995039 19718 consensus_queue.cc:1035] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16cbd1ecc23943a395bb11d12a1b45f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:40:52.012559 19726 mvcc.cc:205] Tried to move back new op lower bound from 7298757640146800640 to 7298757639926054912. Current Snapshot: MvccSnapshot[applied={T|T < 7298757640146800640}]
I20260620 02:40:52.032430 19244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:40:52.032876 19244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:40:52.038708 19244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c91ecc342404860917d19ca467b7f3a: Sending DeleteTablet for 3 replicas of tablet 4a3aeb6ecc7f49d9a88d9c60858516d1
I20260620 02:40:52.040823 19497 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a3aeb6ecc7f49d9a88d9c60858516d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:52 UTC) from {username='slave'} at 127.0.0.1:49194
I20260620 02:40:52.040823 19628 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a3aeb6ecc7f49d9a88d9c60858516d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:52 UTC) from {username='slave'} at 127.0.0.1:45988
I20260620 02:40:52.041098 19366 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a3aeb6ecc7f49d9a88d9c60858516d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:52 UTC) from {username='slave'} at 127.0.0.1:55336
I20260620 02:40:52.046262 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 19304
I20260620 02:40:52.056494 19735 tablet_replica.cc:330] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: stopping tablet replica
I20260620 02:40:52.058095 19735 raft_consensus.cc:2229] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:40:52.058255 19736 tablet_replica.cc:330] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: stopping tablet replica
I20260620 02:40:52.058704 19737 tablet_replica.cc:330] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07: stopping tablet replica
I20260620 02:40:52.058764 19736 raft_consensus.cc:2229] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:40:52.058926 19735 raft_consensus.cc:2258] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:52.059396 19736 raft_consensus.cc:2258] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:52.060056 19737 raft_consensus.cc:2229] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:40:52.062285 19736 ts_tablet_manager.cc:1905] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:52.063637 19735 ts_tablet_manager.cc:1905] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:52.070154 19736 ts_tablet_manager.cc:1918] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:52.070386 19736 log.cc:1196] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/4a3aeb6ecc7f49d9a88d9c60858516d1
I20260620 02:40:52.070967 19736 ts_tablet_manager.cc:1939] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P 16cbd1ecc23943a395bb11d12a1b45f5: Deleting consensus metadata
I20260620 02:40:52.071527 19737 raft_consensus.cc:2258] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:52.071732 19735 ts_tablet_manager.cc:1918] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:52.071954 19735 log.cc:1196] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/4a3aeb6ecc7f49d9a88d9c60858516d1
I20260620 02:40:52.072701 19735 ts_tablet_manager.cc:1939] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c34d28852af44e128660232174ada1fa: Deleting consensus metadata
I20260620 02:40:52.073486 19231 catalog_manager.cc:4841] TS 16cbd1ecc23943a395bb11d12a1b45f5 (127.0.97.2:37327): tablet 4a3aeb6ecc7f49d9a88d9c60858516d1 (table test-table [id=30b71add69e64bd193a71c6a023986bb]) successfully deleted
I20260620 02:40:52.074944 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:52.049916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.0.97.1:34007)
0620 02:40:52.050242 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:52.050251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:52.063908 (+ 13657us) server_negotiation.cc:407] Connection header received
0620 02:40:52.064516 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:52.064523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:52.064655 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:52.064735 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:52.065606 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:52.066930 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:52.069057 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:52.069786 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:52.069902 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:52.072664 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:52.072679 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:52.072703 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:52.073600 (+ 897us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:52.073671 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:52.073700 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:52.073819 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:52.074100 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 02:40:52.074234 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260620 02:40:52.074900 19230 catalog_manager.cc:4841] TS c34d28852af44e128660232174ada1fa (127.0.97.3:40911): tablet 4a3aeb6ecc7f49d9a88d9c60858516d1 (table test-table [id=30b71add69e64bd193a71c6a023986bb]) successfully deleted
I20260620 02:40:52.076025 19426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:52.076572 19737 ts_tablet_manager.cc:1905] T 4a3aeb6ecc7f49d9a88d9c60858516d1 P c52432bf062049b680b8c9f573c1aa07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:52.199234 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 19304
W20260620 02:40:52.220461 19231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34007 (error 108)
I20260620 02:40:52.221280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 19435
I20260620 02:40:52.223487 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:52.222885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34007 (local address 127.0.0.1:55364)
0620 02:40:52.223117 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:52.223276 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260620 02:40:52.224117 19231 catalog_manager.cc:4584] TS c52432bf062049b680b8c9f573c1aa07 (127.0.97.1:34007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a3aeb6ecc7f49d9a88d9c60858516d1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34007: connect: Connection refused (error 111)
I20260620 02:40:52.230862 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:52.222656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.0.97.2:37327)
0620 02:40:52.222893 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:52.222899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:52.222924 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:40:52.222994 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:52.222999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:52.223106 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:52.223202 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:52.224198 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:52.225720 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:52.227348 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:52.227779 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:52.227993 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:52.229254 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:52.229265 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:52.229281 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:52.229958 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:52.230015 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:52.230037 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:52.230187 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:52.230337 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 02:40:52.230461 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:40:52.232213 19557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:52.279261 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:52.278175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34007 (local address 127.0.0.1:55366)
0620 02:40:52.278709 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:52.278875 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260620 02:40:52.281038 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:52.280431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34007 (local address 127.0.0.1:55382)
0620 02:40:52.280687 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:52.280815 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 02:40:52.342458 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:52.341772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34007 (local address 127.0.0.1:55390)
0620 02:40:52.342077 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:52.342218 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 02:40:52.343890 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:52.343369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34007 (local address 127.0.0.1:55400)
0620 02:40:52.343546 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:52.343674 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 02:40:52.361382 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 19435
I20260620 02:40:52.385399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 19566
I20260620 02:40:52.396029 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:52.387193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.0.97.3:40911)
0620 02:40:52.387472 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:52.387477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:52.387497 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:40:52.387580 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:52.387585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:52.387702 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:52.387847 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:52.389395 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:52.390411 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:52.392579 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:52.393029 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:52.393199 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:52.394453 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:52.394463 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:52.394479 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:52.395101 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:52.395154 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:52.395185 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:52.395334 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:52.395479 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 02:40:52.395602 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 02:40:52.397277 19688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:52.425436 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:52.424522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34007 (local address 127.0.0.1:55410)
0620 02:40:52.425068 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:52.425190 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383}
I20260620 02:40:52.427227 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:52.426393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34007 (local address 127.0.0.1:55422)
0620 02:40:52.426931 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:52.427048 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260620 02:40:52.504360 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 19566
I20260620 02:40:52.527930 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 19213
I20260620 02:40:52.530028 19274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:52.668792 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 19213
2026-06-20T02:40:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (4061 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:40:52 dist-test-slave-rpg6 krb5kdc[19746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:40:52 dist-test-slave-rpg6 krb5kdc[19746](info): set up 2 sockets
Jun 20 02:40:52 dist-test-slave-rpg6 krb5kdc[19746](info): commencing operation
krb5kdc: starting...
W20260620 02:40:54.025133 388 mini_kdc.cc:121] Time spent starting KDC: real 1.306s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:40:54 dist-test-slave-rpg6 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:40:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:40:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:40:54.414405 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34271
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:40:54.880299 19759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:54.881048 19759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:54.881242 19759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:54.892962 19759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:40:54.893181 19759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:54.893242 19759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:54.893306 19759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:40:54.893360 19759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:40:54.906450 19759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:54.908089 19759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:54.909642 19759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:54.926335 19764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:54.926431 19765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:54.926935 19767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:54.927544 19759 server_base.cc:1034] running on GCE node
I20260620 02:40:54.929033 19759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:54.931438 19759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:54.935178 19759 hybrid_clock.cc:648] HybridClock initialized: now 1781923254935009 us; error 138 us; skew 500 ppm
I20260620 02:40:54.935999 19759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:54 dist-test-slave-rpg6 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:54.941196 19759 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:40:54.943431 19759 webserver.cc:460] Webserver started at http://127.0.97.62:46531/ using document root <none> and password file <none>
I20260620 02:40:54.944427 19759 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:54.944561 19759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:54.944972 19759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:54.949139 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "70b0980ccaa941e8ba3c588eaa9da445"
format_stamp: "Formatted at 2026-06-20 02:40:54 on dist-test-slave-rpg6"
server_key: "d869dc3ee56474f6012cb5fb2722fe62"
server_key_iv: "49a9ac72c0111199d4419f7cb09c0980"
server_key_version: "encryptionkey@0"
I20260620 02:40:54.950750 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "70b0980ccaa941e8ba3c588eaa9da445"
format_stamp: "Formatted at 2026-06-20 02:40:54 on dist-test-slave-rpg6"
server_key: "d869dc3ee56474f6012cb5fb2722fe62"
server_key_iv: "49a9ac72c0111199d4419f7cb09c0980"
server_key_version: "encryptionkey@0"
I20260620 02:40:54.960813 19759 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.004s
I20260620 02:40:54.967520 19774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:54.970075 19759 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:40:54.970458 19759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "70b0980ccaa941e8ba3c588eaa9da445"
format_stamp: "Formatted at 2026-06-20 02:40:54 on dist-test-slave-rpg6"
server_key: "d869dc3ee56474f6012cb5fb2722fe62"
server_key_iv: "49a9ac72c0111199d4419f7cb09c0980"
server_key_version: "encryptionkey@0"
I20260620 02:40:54.970778 19759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:55.004050 19759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:55.010535 19759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:55.010936 19759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:55.039651 19825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34271 every 8 connection(s)
I20260620 02:40:55.039958 19759 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34271
I20260620 02:40:55.042404 19759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:40:55.046517 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 19759
I20260620 02:40:55.046759 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:40:55.047386 388 external_mini_cluster.cc:1428] Setting key f243f614cf4e5edc2b069fd10d08d448
I20260620 02:40:55.049340 19826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:55.067538 19826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445: Bootstrap starting.
I20260620 02:40:55.072402 19826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445: Neither blocks nor log segments found. Creating new log.
Jun 20 02:40:55 dist-test-slave-rpg6 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:40:55.073779 19826 log.cc:824] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:55.077522 19826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445: No bootstrap required, opened a new log
I20260620 02:40:55.084410 19829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:55.050621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.0.97.62:34271)
0620 02:40:55.051557 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:55.051571 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:55.051612 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:40:55.053478 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:55.053529 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:55.054316 (+ 787us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:55.054942 (+ 626us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:55.058062 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:55.060364 (+ 2302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:55.062541 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:55.063606 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:55.064006 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:55.074627 (+ 10621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:55.074640 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:55.074659 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:55.074711 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:55.078384 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:55.079298 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:55.079315 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:55.079326 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:55.079535 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:55.079991 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:55.080007 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:55.080010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:55.080712 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:55.081106 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:55.082252 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0620 02:40:55.083041 (+ 789us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":440,"threads_started":1}
I20260620 02:40:55.088806 19826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b0980ccaa941e8ba3c588eaa9da445" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } }
I20260620 02:40:55.089291 19826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:55.089449 19826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b0980ccaa941e8ba3c588eaa9da445, State: Initialized, Role: FOLLOWER
I20260620 02:40:55.090119 19826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b0980ccaa941e8ba3c588eaa9da445" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } }
I20260620 02:40:55.090394 19826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:40:55.090550 19826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:40:55.090751 19826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:55.094034 19826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b0980ccaa941e8ba3c588eaa9da445" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } }
I20260620 02:40:55.094841 19826 leader_election.cc:302] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70b0980ccaa941e8ba3c588eaa9da445; no voters:
I20260620 02:40:55.097046 19831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:40:55.097271 19826 leader_election.cc:288] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:40:55.100072 19831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [term 1 LEADER]: Becoming Leader. State: Replica: 70b0980ccaa941e8ba3c588eaa9da445, State: Running, Role: LEADER
I20260620 02:40:55.100936 19831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b0980ccaa941e8ba3c588eaa9da445" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } }
I20260620 02:40:55.102602 19826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:40:55.105610 19832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70b0980ccaa941e8ba3c588eaa9da445. Latest consensus state: current_term: 1 leader_uuid: "70b0980ccaa941e8ba3c588eaa9da445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b0980ccaa941e8ba3c588eaa9da445" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } } }
I20260620 02:40:55.106781 19832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:55.105588 19833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70b0980ccaa941e8ba3c588eaa9da445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b0980ccaa941e8ba3c588eaa9da445" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } } }
I20260620 02:40:55.110793 19833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445 [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:55.114727 19836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:40:55.124769 19836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:40:55.146217 19836 catalog_manager.cc:1347] Generated new cluster ID: 859976b867984b0fb67cb11798d66ec4
I20260620 02:40:55.146447 19836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:40:55.166026 19836 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:40:55.168326 19836 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:40:55.191356 19836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445: Generated new TSK 0
I20260620 02:40:55.192436 19836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:40:55.281530 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34271
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:40:55.750370 19852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:55.750689 19852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:55.750799 19852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:55.760378 19852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:55.760566 19852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:55.760712 19852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:40:55.772903 19852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:55.774979 19852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:55.777451 19852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:55.795595 19857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:55.795733 19860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:55.796130 19858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:55.797827 19852 server_base.cc:1034] running on GCE node
I20260620 02:40:55.799084 19852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:55.801263 19852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:55.802678 19852 hybrid_clock.cc:648] HybridClock initialized: now 1781923255802548 us; error 130 us; skew 500 ppm
I20260620 02:40:55.803611 19852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:55 dist-test-slave-rpg6 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:55.808652 19852 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:40:55.811141 19852 webserver.cc:460] Webserver started at http://127.0.97.1:45859/ using document root <none> and password file <none>
I20260620 02:40:55.812364 19852 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:55.812584 19852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:55.813026 19852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:55.817520 19852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "6efe70589f6c46b2a0e5024eb76b81b0"
format_stamp: "Formatted at 2026-06-20 02:40:55 on dist-test-slave-rpg6"
server_key: "42c851ab1a038de35e8ace5bde6ed72c"
server_key_iv: "d5c621d657a4b9ce34249cf59be1a307"
server_key_version: "encryptionkey@0"
I20260620 02:40:55.818737 19852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6efe70589f6c46b2a0e5024eb76b81b0"
format_stamp: "Formatted at 2026-06-20 02:40:55 on dist-test-slave-rpg6"
server_key: "42c851ab1a038de35e8ace5bde6ed72c"
server_key_iv: "d5c621d657a4b9ce34249cf59be1a307"
server_key_version: "encryptionkey@0"
I20260620 02:40:55.827591 19852 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260620 02:40:55.833076 19867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:55.835623 19852 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260620 02:40:55.835991 19852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "6efe70589f6c46b2a0e5024eb76b81b0"
format_stamp: "Formatted at 2026-06-20 02:40:55 on dist-test-slave-rpg6"
server_key: "42c851ab1a038de35e8ace5bde6ed72c"
server_key_iv: "d5c621d657a4b9ce34249cf59be1a307"
server_key_version: "encryptionkey@0"
I20260620 02:40:55.836288 19852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:55.855178 19852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:55.861622 19852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:55.862028 19852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:55.863508 19852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:55.865869 19852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:55.866004 19852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:55.866173 19852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:55.866250 19852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:55.917303 19852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42701
I20260620 02:40:55.917330 19979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42701 every 8 connection(s)
I20260620 02:40:55.919224 19852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:40:55.925154 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 19852
I20260620 02:40:55.925443 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:40:55.926323 388 external_mini_cluster.cc:1428] Setting key 68e27b813029a7c974a0e471f444fd06
Jun 20 02:40:55 dist-test-slave-rpg6 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:40:55.949595 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:55.923142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45311 (local address 127.0.97.62:34271)
0620 02:40:55.924012 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:55.924018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:55.925749 (+ 1731us) server_negotiation.cc:407] Connection header received
0620 02:40:55.929564 (+ 3815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:55.929574 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:55.929724 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:55.929937 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:55.932399 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:55.933675 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:55.935939 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:55.936800 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:55.936998 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:55.942232 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:55.942240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:55.942245 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:55.942288 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:55.945280 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:55.946225 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:55.946230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:55.946233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:55.946404 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:55.946898 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:55.946908 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:55.946910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:55.947201 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:55.947375 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:55.948825 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0620 02:40:55.949178 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":504,"threads_started":1}
I20260620 02:40:55.951241 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:55.923769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34271 (local address 127.0.97.1:45311)
0620 02:40:55.925484 (+ 1715us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:55.925550 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:55.928549 (+ 2999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:55.930321 (+ 1772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:55.930332 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:55.931140 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:55.932101 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:55.932129 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:55.933935 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:55.933940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:55.935674 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:55.935695 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:55.937337 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:55.937368 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:55.938134 (+ 766us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:55.939048 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:40:55.939077 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:55.941890 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:55.945505 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:55.945516 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:55.945544 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:55.946050 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:55.946534 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:55.946540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:55.946544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:55.946765 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:55.947344 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:55.947354 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:55.947793 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 02:40:55.948841 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0620 02:40:55.949550 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"thread_start_us":619,"threads_started":1}
I20260620 02:40:55.954654 19980 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34271
I20260620 02:40:55.955219 19980 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:55.956619 19980 heartbeater.cc:511] Master 127.0.97.62:34271 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:40:55.960335 19791 ts_manager.cc:194] Registered new tserver with Master: 6efe70589f6c46b2a0e5024eb76b81b0 (127.0.97.1:42701)
I20260620 02:40:55.962517 19791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:40:56.005515 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34271
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:40:56.487807 19986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:56.488224 19986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:56.488385 19986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:56.496368 19986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:56.496485 19986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:56.496659 19986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:40:56.508430 19986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:56.509965 19986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:56.511469 19986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:56.530851 19994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:56.530814 19992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:56.530984 19991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:56.532088 19986 server_base.cc:1034] running on GCE node
I20260620 02:40:56.532863 19986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:56.534075 19986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:56.535627 19986 hybrid_clock.cc:648] HybridClock initialized: now 1781923256535350 us; error 75 us; skew 500 ppm
I20260620 02:40:56.536103 19986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:56 dist-test-slave-rpg6 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:56.541579 19986 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:40:56.544070 19986 webserver.cc:460] Webserver started at http://127.0.97.2:44505/ using document root <none> and password file <none>
I20260620 02:40:56.544987 19986 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:56.545169 19986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:56.545537 19986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:56.549178 19986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "85db4b438dd44c7491de445d18cdd213"
format_stamp: "Formatted at 2026-06-20 02:40:56 on dist-test-slave-rpg6"
server_key: "7f47757e07a8f4ccca4ced246598047d"
server_key_iv: "c342c8d41aa2c3a9923cb5d871e59b98"
server_key_version: "encryptionkey@0"
I20260620 02:40:56.550168 19986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "85db4b438dd44c7491de445d18cdd213"
format_stamp: "Formatted at 2026-06-20 02:40:56 on dist-test-slave-rpg6"
server_key: "7f47757e07a8f4ccca4ced246598047d"
server_key_iv: "c342c8d41aa2c3a9923cb5d871e59b98"
server_key_version: "encryptionkey@0"
I20260620 02:40:56.558892 19986 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260620 02:40:56.566044 20001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:56.568955 19986 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.002s
I20260620 02:40:56.569270 19986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "85db4b438dd44c7491de445d18cdd213"
format_stamp: "Formatted at 2026-06-20 02:40:56 on dist-test-slave-rpg6"
server_key: "7f47757e07a8f4ccca4ced246598047d"
server_key_iv: "c342c8d41aa2c3a9923cb5d871e59b98"
server_key_version: "encryptionkey@0"
I20260620 02:40:56.569573 19986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:56.586812 19986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:56.593999 19986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:56.594400 19986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:56.595562 19986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:56.597270 19986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:56.597390 19986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:56.597548 19986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:56.597623 19986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:56.648840 19986 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36485
I20260620 02:40:56.649082 20113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36485 every 8 connection(s)
I20260620 02:40:56.650715 19986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:40:56.660070 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 19986
I20260620 02:40:56.660292 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:40:56.661286 388 external_mini_cluster.cc:1428] Setting key 556d5f542d82dee6e066c70e4fb22e57
Jun 20 02:40:56 dist-test-slave-rpg6 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:40:56.691778 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:56.654563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44873 (local address 127.0.97.62:34271)
0620 02:40:56.655292 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:56.655296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:56.656293 (+ 997us) server_negotiation.cc:407] Connection header received
0620 02:40:56.659113 (+ 2820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:56.659119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:56.659238 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:56.659383 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:56.662905 (+ 3522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:56.664131 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:56.668462 (+ 4331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:56.669487 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:56.669651 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:56.677908 (+ 8257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:56.677918 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:56.677924 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:56.677984 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:56.685474 (+ 7490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:56.686933 (+ 1459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:56.686943 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:56.686949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:56.687180 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:56.688171 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:56.688185 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:56.688191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:56.688721 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:56.688919 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:56.690876 (+ 1957us) server_negotiation.cc:299] Negotiation successful
0620 02:40:56.691230 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":505,"threads_started":1}
I20260620 02:40:56.693198 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:56.655193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34271 (local address 127.0.97.2:44873)
0620 02:40:56.656112 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:56.656154 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:56.658063 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:56.659676 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:56.659686 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:56.661001 (+ 1315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:56.662645 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:56.662677 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:56.664456 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:56.664463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:56.668021 (+ 3558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:56.668067 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:56.669990 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:56.670006 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:56.670445 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:56.671605 (+ 1160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:40:56.671641 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:56.677439 (+ 5798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:56.685828 (+ 8389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:56.685838 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:56.685857 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:56.686694 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:56.687601 (+ 907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:56.687606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:56.687608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:56.687898 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:56.689108 (+ 1210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:56.689119 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:56.690172 (+ 1053us) client_negotiation.cc:769] Sending connection context
0620 02:40:56.690626 (+ 454us) client_negotiation.cc:240] Negotiation successful
0620 02:40:56.691319 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":310,"threads_started":1}
I20260620 02:40:56.696923 20114 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34271
I20260620 02:40:56.697371 20114 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:56.698262 20114 heartbeater.cc:511] Master 127.0.97.62:34271 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:40:56.701068 19791 ts_manager.cc:194] Registered new tserver with Master: 85db4b438dd44c7491de445d18cdd213 (127.0.97.2:36485)
I20260620 02:40:56.702430 19791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:40:56.753997 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34271
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:40:56.968108 19980 heartbeater.cc:503] Master 127.0.97.62:34271 was elected leader, sending a full tablet report...
W20260620 02:40:57.198123 20120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:57.198622 20120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:57.198807 20120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:57.208065 20120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:57.208281 20120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:57.208474 20120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:40:57.222688 20120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:57.225764 20120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:57.227793 20120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:57.252614 20126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:57.252491 20125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:57.252693 20128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:57.253561 20120 server_base.cc:1034] running on GCE node
I20260620 02:40:57.254408 20120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:57.256093 20120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:57.257542 20120 hybrid_clock.cc:648] HybridClock initialized: now 1781923257257345 us; error 194 us; skew 500 ppm
I20260620 02:40:57.258157 20120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:40:57 dist-test-slave-rpg6 krb5kdc[19746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:40:57.263814 20120 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:40:57.266618 20120 webserver.cc:460] Webserver started at http://127.0.97.3:40787/ using document root <none> and password file <none>
I20260620 02:40:57.267514 20120 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:57.267701 20120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:57.268121 20120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:57.271531 20120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "42660318f55f4e0495822e92e71d2bc9"
format_stamp: "Formatted at 2026-06-20 02:40:57 on dist-test-slave-rpg6"
server_key: "8e25c1de67eb9b77dc319437b9564af0"
server_key_iv: "7a4f5573afc6ed25550b8ff78dede250"
server_key_version: "encryptionkey@0"
I20260620 02:40:57.272716 20120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "42660318f55f4e0495822e92e71d2bc9"
format_stamp: "Formatted at 2026-06-20 02:40:57 on dist-test-slave-rpg6"
server_key: "8e25c1de67eb9b77dc319437b9564af0"
server_key_iv: "7a4f5573afc6ed25550b8ff78dede250"
server_key_version: "encryptionkey@0"
I20260620 02:40:57.280247 20120 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 02:40:57.286885 20135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:57.289759 20120 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:40:57.290234 20120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "42660318f55f4e0495822e92e71d2bc9"
format_stamp: "Formatted at 2026-06-20 02:40:57 on dist-test-slave-rpg6"
server_key: "8e25c1de67eb9b77dc319437b9564af0"
server_key_iv: "7a4f5573afc6ed25550b8ff78dede250"
server_key_version: "encryptionkey@0"
I20260620 02:40:57.290748 20120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:57.334414 20120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:57.343508 20120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:57.343973 20120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:57.345667 20120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:57.348757 20120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:57.348872 20120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:57.348975 20120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:57.349011 20120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:57.400835 20120 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38493
I20260620 02:40:57.400871 20247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38493 every 8 connection(s)
I20260620 02:40:57.402354 20120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:40:57.408752 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 20120
I20260620 02:40:57.408993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:40:57.409794 388 external_mini_cluster.cc:1428] Setting key a40febf44dc1b15df61bbe1d937c60da
Jun 20 02:40:57 dist-test-slave-rpg6 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:40:57.435925 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.407020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50499 (local address 127.0.97.62:34271)
0620 02:40:57.408151 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:57.408156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:57.410987 (+ 2831us) server_negotiation.cc:407] Connection header received
0620 02:40:57.414110 (+ 3123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:57.414118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:57.414301 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:57.414638 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:57.418227 (+ 3589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.419677 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.422004 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.422526 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.422649 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.428936 (+ 6287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:57.428942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:57.428945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:57.428994 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:57.432160 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:57.433146 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:57.433150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:57.433153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:57.433289 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:57.433798 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:57.433803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:57.433806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:57.434131 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:57.434312 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:57.435174 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 02:40:57.435466 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":542,"threads_started":1}
I20260620 02:40:57.437157 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.408530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34271 (local address 127.0.97.3:50499)
0620 02:40:57.410127 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:57.410427 (+ 300us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:57.413616 (+ 3189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:57.415134 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:57.415149 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:57.416432 (+ 1283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:57.417745 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.417782 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.420123 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.420129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.421596 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.421618 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.422861 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.422868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.423138 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.424701 (+ 1563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:40:57.424779 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:57.428655 (+ 3876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:57.432393 (+ 3738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:57.432404 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:57.432430 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:57.432973 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:57.433416 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:40:57.433422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:40:57.433425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:40:57.433655 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:40:57.434278 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:57.434288 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:57.434736 (+ 448us) client_negotiation.cc:769] Sending connection context
0620 02:40:57.435068 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 02:40:57.435714 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":586,"threads_started":1}
I20260620 02:40:57.439432 20248 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34271
I20260620 02:40:57.439795 20248 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:57.440940 20248 heartbeater.cc:511] Master 127.0.97.62:34271 requested a full tablet report, sending...
I20260620 02:40:57.443667 19790 ts_manager.cc:194] Registered new tserver with Master: 42660318f55f4e0495822e92e71d2bc9 (127.0.97.3:38493)
I20260620 02:40:57.445022 19790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50499
I20260620 02:40:57.457857 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:40:57.482671 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.464366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35958 (local address 127.0.97.62:34271)
0620 02:40:57.464735 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:57.464740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:57.465059 (+ 319us) server_negotiation.cc:407] Connection header received
0620 02:40:57.465228 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:57.465233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:57.465315 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:57.465524 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:57.467284 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.468241 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.470167 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.470694 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.470936 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.473978 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:57.474004 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:57.474011 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:57.474071 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:57.477621 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:57.478644 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:57.478649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:57.478680 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:57.479073 (+ 393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:57.479615 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:57.479621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:57.479624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:57.480380 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:57.480879 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:57.481677 (+ 798us) server_negotiation.cc:299] Negotiation successful
0620 02:40:57.482186 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 02:40:57.489452 19790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:40:57.497171 19790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:40:57.538203 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.514108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.0.97.2:36485)
0620 02:40:57.528552 (+ 14444us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:57.528559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:57.528586 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:40:57.528677 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:57.528683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:57.528941 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:57.529175 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:57.530758 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.532279 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.535463 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.536605 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.536789 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.537041 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:57.537613 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 02:40:57.537853 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14281,"thread_start_us":14170,"threads_started":1}
I20260620 02:40:57.538251 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.513918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36485 (local address 127.0.0.1:55466)
0620 02:40:57.526532 (+ 12614us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:57.526590 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:57.526848 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:57.529201 (+ 2353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:57.529210 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:57.529258 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:57.529788 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.529804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.532508 (+ 2704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.532516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.535167 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.535188 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.536880 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.536887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.537258 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.537370 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 02:40:57.537529 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 02:40:57.537791 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12229,"mutex_wait_us":101,"thread_start_us":16776,"threads_started":1}
I20260620 02:40:57.541152 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.528109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.0.97.3:38493)
0620 02:40:57.530440 (+ 2331us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:57.530449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:57.530957 (+ 508us) server_negotiation.cc:407] Connection header received
0620 02:40:57.531077 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:57.531086 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:57.531393 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:57.531647 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:57.533500 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.534893 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.537939 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.539514 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.539653 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.539949 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:57.540591 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 02:40:57.540768 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2080,"thread_start_us":1854,"threads_started":1}
I20260620 02:40:57.541132 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.529749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38493 (local address 127.0.0.1:56452)
0620 02:40:57.530743 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:57.530762 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:57.530910 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:57.532425 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:57.532429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:57.532561 (+ 132us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:57.533246 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.533258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.535200 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.535210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.537722 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.537736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.540128 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.540134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.540275 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.540399 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 02:40:57.540517 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 02:40:57.540678 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":105,"spinlock_wait_cycles":11008,"thread_start_us":660,"threads_started":1}
I20260620 02:40:57.541656 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.514386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.0.97.1:42701)
0620 02:40:57.527888 (+ 13502us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:57.527899 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:57.530744 (+ 2845us) server_negotiation.cc:407] Connection header received
0620 02:40:57.531195 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:57.531202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:57.531414 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:57.531647 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:57.532414 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.534336 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.537424 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.539607 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.539749 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.539981 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:57.540736 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 02:40:57.540932 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13261,"thread_start_us":13516,"threads_started":1}
I20260620 02:40:57.542557 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.514076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42701 (local address 127.0.0.1:52630)
0620 02:40:57.530588 (+ 16512us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:57.530612 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:57.530910 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:57.531637 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:57.531642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:57.531670 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:57.532173 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.532189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.534517 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.534523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.537187 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.537208 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.540210 (+ 3002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.540221 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.540507 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.540552 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 02:40:57.541931 (+ 1379us) client_negotiation.cc:240] Negotiation successful
0620 02:40:57.542102 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16320,"mutex_wait_us":23,"spinlock_wait_cycles":3712,"thread_start_us":12325,"threads_started":1}
I20260620 02:40:57.543563 20049 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab08296046b4526b32dde411f9bfbe2 (DEFAULT_TABLE table=test-table [id=552ed9008c22486e8d6192b482c0c076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:57.545017 20049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab08296046b4526b32dde411f9bfbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:57.545456 20183 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab08296046b4526b32dde411f9bfbe2 (DEFAULT_TABLE table=test-table [id=552ed9008c22486e8d6192b482c0c076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:57.547363 20183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab08296046b4526b32dde411f9bfbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:57.547186 19915 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab08296046b4526b32dde411f9bfbe2 (DEFAULT_TABLE table=test-table [id=552ed9008c22486e8d6192b482c0c076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:40:57.548985 19915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab08296046b4526b32dde411f9bfbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:57.561865 20267 tablet_bootstrap.cc:492] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: Bootstrap starting.
I20260620 02:40:57.567174 20267 tablet_bootstrap.cc:654] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:57.568625 20267 log.cc:824] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:57.572507 20267 tablet_bootstrap.cc:492] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: No bootstrap required, opened a new log
I20260620 02:40:57.572741 20268 tablet_bootstrap.cc:492] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: Bootstrap starting.
I20260620 02:40:57.573105 20267 ts_tablet_manager.cc:1397] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260620 02:40:57.577426 20268 tablet_bootstrap.cc:654] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:57.578450 20270 tablet_bootstrap.cc:492] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: Bootstrap starting.
I20260620 02:40:57.579337 20268 log.cc:824] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:57.583500 20270 tablet_bootstrap.cc:654] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:57.583818 20268 tablet_bootstrap.cc:492] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: No bootstrap required, opened a new log
I20260620 02:40:57.584203 20268 ts_tablet_manager.cc:1397] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260620 02:40:57.584372 20267 raft_consensus.cc:348] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db4b438dd44c7491de445d18cdd213" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36485 } } peers { permanent_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42701 } } peers { permanent_uuid: "42660318f55f4e0495822e92e71d2bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38493 } }
I20260620 02:40:57.584915 20267 raft_consensus.cc:374] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:57.585019 20267 raft_consensus.cc:729] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85db4b438dd44c7491de445d18cdd213, State: Initialized, Role: FOLLOWER
I20260620 02:40:57.585467 20270 log.cc:824] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:57.585742 20267 consensus_queue.cc:260] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db4b438dd44c7491de445d18cdd213" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36485 } } peers { permanent_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42701 } } peers { permanent_uuid: "42660318f55f4e0495822e92e71d2bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38493 } }
I20260620 02:40:57.594496 20267 ts_tablet_manager.cc:1428] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260620 02:40:57.594946 20114 heartbeater.cc:503] Master 127.0.97.62:34271 was elected leader, sending a full tablet report...
I20260620 02:40:57.597153 20270 tablet_bootstrap.cc:492] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: No bootstrap required, opened a new log
I20260620 02:40:57.597509 20270 ts_tablet_manager.cc:1397] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.006s
I20260620 02:40:57.598555 20268 raft_consensus.cc:348] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db4b438dd44c7491de445d18cdd213" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36485 } } peers { permanent_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42701 } } peers { permanent_uuid: "42660318f55f4e0495822e92e71d2bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38493 } }
I20260620 02:40:57.599004 20268 raft_consensus.cc:374] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:57.599109 20268 raft_consensus.cc:729] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42660318f55f4e0495822e92e71d2bc9, State: Initialized, Role: FOLLOWER
I20260620 02:40:57.599886 20268 consensus_queue.cc:260] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db4b438dd44c7491de445d18cdd213" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36485 } } peers { permanent_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42701 } } peers { permanent_uuid: "42660318f55f4e0495822e92e71d2bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38493 } }
I20260620 02:40:57.601148 20248 heartbeater.cc:503] Master 127.0.97.62:34271 was elected leader, sending a full tablet report...
I20260620 02:40:57.602001 20268 ts_tablet_manager.cc:1428] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260620 02:40:57.610107 20270 raft_consensus.cc:348] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db4b438dd44c7491de445d18cdd213" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36485 } } peers { permanent_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42701 } } peers { permanent_uuid: "42660318f55f4e0495822e92e71d2bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38493 } }
I20260620 02:40:57.610562 20270 raft_consensus.cc:374] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:57.610632 20270 raft_consensus.cc:729] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe70589f6c46b2a0e5024eb76b81b0, State: Initialized, Role: FOLLOWER
I20260620 02:40:57.611183 20270 consensus_queue.cc:260] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db4b438dd44c7491de445d18cdd213" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36485 } } peers { permanent_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42701 } } peers { permanent_uuid: "42660318f55f4e0495822e92e71d2bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38493 } }
I20260620 02:40:57.613725 20270 ts_tablet_manager.cc:1428] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
W20260620 02:40:57.654290 20115 tablet.cc:2369] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:40:57.654867 20249 tablet.cc:2369] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:40:57.673115 19981 tablet.cc:2369] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:40:57.675406 20274 raft_consensus.cc:482] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:40:57.675812 20274 raft_consensus.cc:504] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db4b438dd44c7491de445d18cdd213" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36485 } } peers { permanent_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42701 } } peers { permanent_uuid: "42660318f55f4e0495822e92e71d2bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38493 } }
I20260620 02:40:57.677896 20274 leader_election.cc:288] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85db4b438dd44c7491de445d18cdd213 (127.0.97.2:36485), 6efe70589f6c46b2a0e5024eb76b81b0 (127.0.97.1:42701)
I20260620 02:40:57.686132 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.678572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59947 (local address 127.0.97.2:36485)
0620 02:40:57.678893 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:57.678896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:57.678914 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:40:57.679080 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:57.679088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:57.679187 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:57.679314 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:57.679767 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.680830 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.683146 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.684904 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.685099 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.685198 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:57.685756 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 02:40:57.685907 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:40:57.686172 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.678484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36485 (local address 127.0.97.3:59947)
0620 02:40:57.678716 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:57.678740 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:57.678891 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:57.679349 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:57.679354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:57.679387 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:57.679602 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.679614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.681052 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.681059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.682847 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.682862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.685189 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.685196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.685513 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.685555 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 02:40:57.685686 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 02:40:57.685859 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 02:40:57.687987 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.678484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42701 (local address 127.0.97.3:47721)
0620 02:40:57.679624 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:57.679640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:57.679789 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:57.680229 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:57.680234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:57.680269 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:40:57.680591 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.680601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.682351 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.682359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.684946 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:57.684967 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.687168 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.687173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:57.687322 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.687356 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:40:57.687451 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:40:57.687594 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"mutex_wait_us":255,"thread_start_us":429,"threads_started":1}
I20260620 02:40:57.687996 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.678572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47721 (local address 127.0.97.1:42701)
0620 02:40:57.678803 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:57.678809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:57.679775 (+ 966us) server_negotiation.cc:407] Connection header received
0620 02:40:57.680034 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:57.680038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:57.680100 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:57.680173 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:40:57.680822 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.682019 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.685182 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.686934 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.687063 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.687189 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:57.687518 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 02:40:57.687687 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 02:40:57.687777 20069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab08296046b4526b32dde411f9bfbe2" candidate_uuid: "42660318f55f4e0495822e92e71d2bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85db4b438dd44c7491de445d18cdd213" is_pre_election: true
I20260620 02:40:57.688537 20069 raft_consensus.cc:2454] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42660318f55f4e0495822e92e71d2bc9 in term 0.
I20260620 02:40:57.689198 19935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab08296046b4526b32dde411f9bfbe2" candidate_uuid: "42660318f55f4e0495822e92e71d2bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" is_pre_election: true
I20260620 02:40:57.689483 20138 leader_election.cc:302] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42660318f55f4e0495822e92e71d2bc9, 85db4b438dd44c7491de445d18cdd213; no voters:
I20260620 02:40:57.689893 19935 raft_consensus.cc:2454] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42660318f55f4e0495822e92e71d2bc9 in term 0.
I20260620 02:40:57.689966 20274 raft_consensus.cc:2781] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:40:57.690114 20274 raft_consensus.cc:482] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:40:57.690225 20274 raft_consensus.cc:3037] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:57.692734 20274 raft_consensus.cc:504] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db4b438dd44c7491de445d18cdd213" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36485 } } peers { permanent_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42701 } } peers { permanent_uuid: "42660318f55f4e0495822e92e71d2bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38493 } }
I20260620 02:40:57.693401 20274 leader_election.cc:288] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 85db4b438dd44c7491de445d18cdd213 (127.0.97.2:36485), 6efe70589f6c46b2a0e5024eb76b81b0 (127.0.97.1:42701)
I20260620 02:40:57.694176 19935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab08296046b4526b32dde411f9bfbe2" candidate_uuid: "42660318f55f4e0495822e92e71d2bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efe70589f6c46b2a0e5024eb76b81b0"
I20260620 02:40:57.694176 20069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab08296046b4526b32dde411f9bfbe2" candidate_uuid: "42660318f55f4e0495822e92e71d2bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85db4b438dd44c7491de445d18cdd213"
I20260620 02:40:57.694546 20069 raft_consensus.cc:3037] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:57.694546 19935 raft_consensus.cc:3037] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:57.698216 20069 raft_consensus.cc:2454] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42660318f55f4e0495822e92e71d2bc9 in term 1.
I20260620 02:40:57.698222 19935 raft_consensus.cc:2454] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42660318f55f4e0495822e92e71d2bc9 in term 1.
I20260620 02:40:57.698844 20139 leader_election.cc:302] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42660318f55f4e0495822e92e71d2bc9, 6efe70589f6c46b2a0e5024eb76b81b0; no voters:
I20260620 02:40:57.699257 20274 raft_consensus.cc:2781] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:40:57.699772 20274 raft_consensus.cc:686] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 42660318f55f4e0495822e92e71d2bc9, State: Running, Role: LEADER
I20260620 02:40:57.700417 20274 consensus_queue.cc:237] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db4b438dd44c7491de445d18cdd213" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36485 } } peers { permanent_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42701 } } peers { permanent_uuid: "42660318f55f4e0495822e92e71d2bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38493 } }
I20260620 02:40:57.707137 19790 catalog_manager.cc:5495] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42660318f55f4e0495822e92e71d2bc9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "42660318f55f4e0495822e92e71d2bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85db4b438dd44c7491de445d18cdd213" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42660318f55f4e0495822e92e71d2bc9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38493 } health_report { overall_health: HEALTHY } } }
I20260620 02:40:57.747104 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.738039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.0.97.3:38493)
0620 02:40:57.738294 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:57.738299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:57.738326 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:40:57.738673 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:57.738681 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:57.739169 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:57.739367 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:40:57.740279 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.741244 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.743711 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.744421 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.744569 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.745303 (+ 734us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:40:57.746267 (+ 964us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:40:57.746608 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:57.746672 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 02:40:57.746874 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:40:57.758222 19935 raft_consensus.cc:1261] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0 [term 1 FOLLOWER]: Refusing update from remote peer 42660318f55f4e0495822e92e71d2bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:40:57.758827 20069 raft_consensus.cc:1261] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213 [term 1 FOLLOWER]: Refusing update from remote peer 42660318f55f4e0495822e92e71d2bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:40:57.760169 20277 consensus_queue.cc:1035] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe70589f6c46b2a0e5024eb76b81b0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:40:57.760953 20274 consensus_queue.cc:1035] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85db4b438dd44c7491de445d18cdd213" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:40:57.775593 20280 mvcc.cc:205] Tried to move back new op lower bound from 7298757663763763200 to 7298757663545806848. Current Snapshot: MvccSnapshot[applied={T|T < 7298757663763763200}]
I20260620 02:40:57.777262 20282 mvcc.cc:205] Tried to move back new op lower bound from 7298757663763763200 to 7298757663545806848. Current Snapshot: MvccSnapshot[applied={T|T < 7298757663763763200}]
I20260620 02:40:57.795634 19790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:40:57.796105 19790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:40:57.801914 19790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70b0980ccaa941e8ba3c588eaa9da445: Sending DeleteTablet for 3 replicas of tablet 7ab08296046b4526b32dde411f9bfbe2
I20260620 02:40:57.803649 20049 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab08296046b4526b32dde411f9bfbe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55466
I20260620 02:40:57.803912 19915 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab08296046b4526b32dde411f9bfbe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52630
I20260620 02:40:57.807077 20183 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab08296046b4526b32dde411f9bfbe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:40:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56452
I20260620 02:40:57.809839 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 19852
I20260620 02:40:57.820623 20290 tablet_replica.cc:330] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: stopping tablet replica
I20260620 02:40:57.821146 20291 tablet_replica.cc:330] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: stopping tablet replica
I20260620 02:40:57.821198 20290 raft_consensus.cc:2229] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:40:57.822052 20291 raft_consensus.cc:2229] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:40:57.822258 20290 raft_consensus.cc:2258] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:57.823781 20291 raft_consensus.cc:2258] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:57.825312 20292 tablet_replica.cc:330] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: stopping tablet replica
I20260620 02:40:57.825819 20292 raft_consensus.cc:2229] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:40:57.826658 20292 raft_consensus.cc:2258] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:40:57.828569 20291 ts_tablet_manager.cc:1905] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:57.828996 20290 ts_tablet_manager.cc:1905] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:40:57 dist-test-slave-rpg6 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:40:57.838044 20292 ts_tablet_manager.cc:1905] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:40:57.838547 20290 ts_tablet_manager.cc:1918] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:57.838764 20290 log.cc:1196] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/7ab08296046b4526b32dde411f9bfbe2
I20260620 02:40:57.839694 20290 ts_tablet_manager.cc:1939] T 7ab08296046b4526b32dde411f9bfbe2 P 85db4b438dd44c7491de445d18cdd213: Deleting consensus metadata
I20260620 02:40:57.840881 20291 ts_tablet_manager.cc:1918] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:57.842588 19777 catalog_manager.cc:4841] TS 85db4b438dd44c7491de445d18cdd213 (127.0.97.2:36485): tablet 7ab08296046b4526b32dde411f9bfbe2 (table test-table [id=552ed9008c22486e8d6192b482c0c076]) successfully deleted
I20260620 02:40:57.843683 20291 log.cc:1196] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/7ab08296046b4526b32dde411f9bfbe2
I20260620 02:40:57.844867 20291 ts_tablet_manager.cc:1939] T 7ab08296046b4526b32dde411f9bfbe2 P 6efe70589f6c46b2a0e5024eb76b81b0: Deleting consensus metadata
I20260620 02:40:57.848564 19778 catalog_manager.cc:4841] TS 6efe70589f6c46b2a0e5024eb76b81b0 (127.0.97.1:42701): tablet 7ab08296046b4526b32dde411f9bfbe2 (table test-table [id=552ed9008c22486e8d6192b482c0c076]) successfully deleted
I20260620 02:40:57.853314 20292 ts_tablet_manager.cc:1918] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:40:57.853814 20292 log.cc:1196] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/7ab08296046b4526b32dde411f9bfbe2
I20260620 02:40:57.854461 20292 ts_tablet_manager.cc:1939] T 7ab08296046b4526b32dde411f9bfbe2 P 42660318f55f4e0495822e92e71d2bc9: Deleting consensus metadata
I20260620 02:40:57.854573 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.811402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.0.97.1:42701)
0620 02:40:57.811970 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:57.811984 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:57.816956 (+ 4972us) server_negotiation.cc:407] Connection header received
0620 02:40:57.817197 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:57.817207 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:57.817397 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:57.817611 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:57.821190 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.823550 (+ 2360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.826591 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.827574 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:57.827783 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:57.841940 (+ 14157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:57.841964 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:57.841987 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:57.842051 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:57.849425 (+ 7374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:57.850920 (+ 1495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:57.850933 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:57.850937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:57.851170 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:57.851746 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:57.851752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:57.851755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:57.852371 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:57.852723 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:57.853726 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0620 02:40:57.854056 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 02:40:57.856514 19975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:57.857880 19778 catalog_manager.cc:4841] TS 42660318f55f4e0495822e92e71d2bc9 (127.0.97.3:38493): tablet 7ab08296046b4526b32dde411f9bfbe2 (table test-table [id=552ed9008c22486e8d6192b482c0c076]) successfully deleted
I20260620 02:40:57.968888 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 19852
I20260620 02:40:57.995272 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 19986
Jun 20 02:40:58 dist-test-slave-rpg6 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:40:58.018818 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:57.996416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.0.97.2:36485)
0620 02:40:57.996660 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:57.996664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:57.996685 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:40:57.996778 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:57.996782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:57.996931 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:57.997038 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:57.999058 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:57.999974 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:58.002016 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:58.002490 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:58.002610 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:58.010082 (+ 7472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:58.010101 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:58.010119 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:58.010183 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:58.015134 (+ 4951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:58.016330 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:58.016341 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:58.016344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:58.016532 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:58.017007 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:58.017011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:58.017013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:58.017435 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:58.017891 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:58.018198 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 02:40:58.018433 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260620 02:40:58.019505 20109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:58.148988 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 19986
I20260620 02:40:58.170315 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 20120
Jun 20 02:40:58 dist-test-slave-rpg6 krb5kdc[19746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:40:58.193463 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:58.171237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.0.97.3:38493)
0620 02:40:58.171491 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:58.171496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:58.171629 (+ 133us) server_negotiation.cc:407] Connection header received
0620 02:40:58.172095 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:58.172099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:58.172172 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:58.172269 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:58.174228 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:58.175278 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:58.177848 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:58.178975 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:58.179252 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:58.186365 (+ 7113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:58.186380 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:58.186397 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:40:58.186452 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:58.190185 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:58.191079 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:58.191090 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:58.191092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:58.191236 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:40:58.191655 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:40:58.191660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:40:58.191663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:40:58.192181 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:58.192548 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:58.192856 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 02:40:58.193096 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 02:40:58.194361 20243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:58.336925 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 20120
I20260620 02:40:58.362735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 19759
I20260620 02:40:58.364509 19821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:40:58.523213 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 19759
2026-06-20T02:40:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5857 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T02:40:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:40:58Z Disabled control of system clock
I20260620 02:40:58.619131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39019
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:40:59.080657 20301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:59.081012 20301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:59.081161 20301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:59.089569 20301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:40:59.089718 20301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:59.089759 20301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:59.089813 20301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:40:59.089850 20301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:40:59.100678 20301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:59.102239 20301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:59.105019 20301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:59.122189 20306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:59.122247 20309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:59.122133 20307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:59.123497 20301 server_base.cc:1034] running on GCE node
I20260620 02:40:59.125000 20301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:59.127957 20301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:59.131541 20301 hybrid_clock.cc:648] HybridClock initialized: now 1781923259131460 us; error 61 us; skew 500 ppm
I20260620 02:40:59.132827 20301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:40:59.136168 20301 webserver.cc:460] Webserver started at http://127.0.97.62:35843/ using document root <none> and password file <none>
I20260620 02:40:59.136972 20301 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:59.137074 20301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:59.137406 20301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:59.140920 20301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "f74632f5f3db44fe9c98a5205418d57a"
format_stamp: "Formatted at 2026-06-20 02:40:59 on dist-test-slave-rpg6"
server_key: "1a584d82ecc2d8f36c8520f00fdd10ca"
server_key_iv: "c6e068d60d42728956945ecb7925af62"
server_key_version: "encryptionkey@0"
I20260620 02:40:59.141927 20301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "f74632f5f3db44fe9c98a5205418d57a"
format_stamp: "Formatted at 2026-06-20 02:40:59 on dist-test-slave-rpg6"
server_key: "1a584d82ecc2d8f36c8520f00fdd10ca"
server_key_iv: "c6e068d60d42728956945ecb7925af62"
server_key_version: "encryptionkey@0"
I20260620 02:40:59.148828 20301 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 02:40:59.153712 20315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:59.156052 20301 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:40:59.156311 20301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "f74632f5f3db44fe9c98a5205418d57a"
format_stamp: "Formatted at 2026-06-20 02:40:59 on dist-test-slave-rpg6"
server_key: "1a584d82ecc2d8f36c8520f00fdd10ca"
server_key_iv: "c6e068d60d42728956945ecb7925af62"
server_key_version: "encryptionkey@0"
I20260620 02:40:59.156577 20301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:59.175652 20301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:59.176816 20301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:59.177105 20301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:59.202917 20301 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39019
I20260620 02:40:59.202940 20366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39019 every 8 connection(s)
I20260620 02:40:59.204523 20301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:40:59.207070 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 20301
I20260620 02:40:59.207299 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:40:59.208052 388 external_mini_cluster.cc:1428] Setting key 307267a8c6e8f2d946af0ada25f73ae0
I20260620 02:40:59.210850 20367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:40:59.226117 20367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a: Bootstrap starting.
I20260620 02:40:59.226667 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:59.210985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.0.97.62:39019)
0620 02:40:59.212401 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:59.212414 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:59.212459 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:40:59.214240 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:59.214268 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:59.214964 (+ 696us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:59.215466 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:59.216196 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:59.217845 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:59.220261 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:59.221124 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:59.221510 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:59.223111 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:59.223120 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:59.223141 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:59.223815 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:59.223993 (+ 178us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:59.224020 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:59.224170 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:59.224835 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 02:40:59.225407 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":509,"threads_started":1}
I20260620 02:40:59.230968 20367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a: Neither blocks nor log segments found. Creating new log.
I20260620 02:40:59.232306 20367 log.cc:824] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:40:59.237510 20367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a: No bootstrap required, opened a new log
I20260620 02:40:59.248446 20367 raft_consensus.cc:348] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74632f5f3db44fe9c98a5205418d57a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39019 } }
I20260620 02:40:59.248962 20367 raft_consensus.cc:374] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:40:59.249050 20367 raft_consensus.cc:729] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74632f5f3db44fe9c98a5205418d57a, State: Initialized, Role: FOLLOWER
I20260620 02:40:59.249782 20367 consensus_queue.cc:260] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74632f5f3db44fe9c98a5205418d57a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39019 } }
I20260620 02:40:59.250089 20367 raft_consensus.cc:388] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:40:59.250252 20367 raft_consensus.cc:482] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:40:59.250597 20367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:40:59.254997 20367 raft_consensus.cc:504] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74632f5f3db44fe9c98a5205418d57a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39019 } }
I20260620 02:40:59.256074 20367 leader_election.cc:302] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f74632f5f3db44fe9c98a5205418d57a; no voters:
I20260620 02:40:59.257290 20367 leader_election.cc:288] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:40:59.257565 20372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:40:59.259521 20372 raft_consensus.cc:686] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [term 1 LEADER]: Becoming Leader. State: Replica: f74632f5f3db44fe9c98a5205418d57a, State: Running, Role: LEADER
I20260620 02:40:59.260412 20372 consensus_queue.cc:237] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74632f5f3db44fe9c98a5205418d57a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39019 } }
I20260620 02:40:59.260583 20367 sys_catalog.cc:564] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:40:59.265375 20374 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74632f5f3db44fe9c98a5205418d57a. Latest consensus state: current_term: 1 leader_uuid: "f74632f5f3db44fe9c98a5205418d57a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74632f5f3db44fe9c98a5205418d57a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39019 } } }
I20260620 02:40:59.267720 20374 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:59.268792 20373 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f74632f5f3db44fe9c98a5205418d57a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74632f5f3db44fe9c98a5205418d57a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39019 } } }
I20260620 02:40:59.269351 20373 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a [sys.catalog]: This master's current role is: LEADER
I20260620 02:40:59.270610 20381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:40:59.279336 20381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:40:59.292428 20381 catalog_manager.cc:1347] Generated new cluster ID: 3d1f69ad379f49158a39e8711953030a
I20260620 02:40:59.292580 20381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:40:59.307179 20381 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:40:59.309064 20381 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:40:59.324133 20381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a: Generated new TSK 0
I20260620 02:40:59.325799 20381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:40:59.333701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39019
--builtin_ntp_servers=127.0.97.20:45199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:40:59.773991 20391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:40:59.774341 20391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:40:59.774473 20391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:40:59.787392 20391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:40:59.787525 20391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:40:59.787711 20391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:40:59.801096 20391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:40:59.802780 20391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:40:59.804894 20391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:40:59.822716 20399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:59.823014 20397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:40:59.823460 20396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:40:59.824303 20391 server_base.cc:1034] running on GCE node
I20260620 02:40:59.825054 20391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:40:59.826393 20391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:40:59.827672 20391 hybrid_clock.cc:648] HybridClock initialized: now 1781923259827627 us; error 45 us; skew 500 ppm
I20260620 02:40:59.828217 20391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:40:59.831103 20391 webserver.cc:460] Webserver started at http://127.0.97.1:34963/ using document root <none> and password file <none>
I20260620 02:40:59.832018 20391 fs_manager.cc:359] Metadata directory not provided
I20260620 02:40:59.832182 20391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:40:59.832546 20391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:40:59.835789 20391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/data/instance:
uuid: "888b69e6bff54b5291af9f59da6bfa06"
format_stamp: "Formatted at 2026-06-20 02:40:59 on dist-test-slave-rpg6"
server_key: "9a5aaf403193276d33fc45bc2e057460"
server_key_iv: "c9aaf32b8532208d69acfbf8373eed8d"
server_key_version: "encryptionkey@0"
I20260620 02:40:59.836822 20391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/wal/instance:
uuid: "888b69e6bff54b5291af9f59da6bfa06"
format_stamp: "Formatted at 2026-06-20 02:40:59 on dist-test-slave-rpg6"
server_key: "9a5aaf403193276d33fc45bc2e057460"
server_key_iv: "c9aaf32b8532208d69acfbf8373eed8d"
server_key_version: "encryptionkey@0"
I20260620 02:40:59.843550 20391 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:40:59.848986 20405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:59.851394 20391 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:40:59.851812 20391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/wal
uuid: "888b69e6bff54b5291af9f59da6bfa06"
format_stamp: "Formatted at 2026-06-20 02:40:59 on dist-test-slave-rpg6"
server_key: "9a5aaf403193276d33fc45bc2e057460"
server_key_iv: "c9aaf32b8532208d69acfbf8373eed8d"
server_key_version: "encryptionkey@0"
I20260620 02:40:59.852247 20391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:40:59.877517 20391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:40:59.878509 20391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:40:59.878834 20391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:40:59.880218 20391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:40:59.882282 20391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:40:59.882427 20391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:59.882578 20391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:40:59.882670 20391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:40:59.937713 20391 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32963
I20260620 02:40:59.937793 20517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32963 every 8 connection(s)
I20260620 02:40:59.939249 20391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:40:59.947245 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 20391
I20260620 02:40:59.947522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:40:59.948240 388 external_mini_cluster.cc:1428] Setting key b070856a1bb90d4719d66f96042f5e4a
I20260620 02:40:59.952437 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39019
--builtin_ntp_servers=127.0.97.20:45199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:40:59.958024 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:59.942446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56789 (local address 127.0.97.62:39019)
0620 02:40:59.943053 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 02:40:59.943058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:40:59.944324 (+ 1266us) server_negotiation.cc:407] Connection header received
0620 02:40:59.946794 (+ 2470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:40:59.946800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:40:59.946916 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:40:59.947055 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:40:59.949887 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:59.950726 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:59.953665 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:40:59.954162 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:40:59.954297 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:59.956425 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:40:59.956431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:40:59.956434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:40:59.956461 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:40:59.956510 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:40:59.956523 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:40:59.956873 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 02:40:59.957508 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 02:40:59.957632 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":269,"threads_started":1}
I20260620 02:40:59.959247 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:40:59.942844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39019 (local address 127.0.97.1:56789)
0620 02:40:59.944140 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0620 02:40:59.944184 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:40:59.946387 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:40:59.947348 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:40:59.947390 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:40:59.947534 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:40:59.949552 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:59.949580 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:59.950932 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:59.950937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:59.953415 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:40:59.953442 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:40:59.954455 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:40:59.954465 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:40:59.954865 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:40:59.955854 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:40:59.955860 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:40:59.955928 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:40:59.956069 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:40:59.956871 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:40:59.956898 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:40:59.957155 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 02:40:59.957462 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 02:40:59.957964 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":349,"threads_started":1}
I20260620 02:40:59.962620 20518 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39019
I20260620 02:40:59.963214 20518 heartbeater.cc:464] Registering TS with master...
I20260620 02:40:59.964639 20518 heartbeater.cc:511] Master 127.0.97.62:39019 requested a full tablet report, sending...
I20260620 02:40:59.967976 20332 ts_manager.cc:194] Registered new tserver with Master: 888b69e6bff54b5291af9f59da6bfa06 (127.0.97.1:32963)
I20260620 02:40:59.969925 20332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:56789
W20260620 02:41:00.403337 20522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:00.403757 20522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:00.403945 20522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:00.412072 20522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:00.412263 20522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:00.412453 20522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:41:00.425671 20522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:00.427276 20522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:00.430217 20522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:00.447521 20527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:00.447877 20530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:00.447942 20528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:00.450129 20522 server_base.cc:1034] running on GCE node
I20260620 02:41:00.451350 20522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:00.453195 20522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:00.454609 20522 hybrid_clock.cc:648] HybridClock initialized: now 1781923260454574 us; error 107 us; skew 500 ppm
I20260620 02:41:00.455159 20522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:41:00.464435 20522 webserver.cc:460] Webserver started at http://127.0.97.2:34479/ using document root <none> and password file <none>
I20260620 02:41:00.465304 20522 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:00.465658 20522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:00.466192 20522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:00.469728 20522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/data/instance:
uuid: "a55cf621b4284cfb93a39fc973efc87d"
format_stamp: "Formatted at 2026-06-20 02:41:00 on dist-test-slave-rpg6"
server_key: "b4da8fa2c8c1a177d2dba607aac072e0"
server_key_iv: "085267f589b2580755f452d7d88e501e"
server_key_version: "encryptionkey@0"
I20260620 02:41:00.470685 20522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a55cf621b4284cfb93a39fc973efc87d"
format_stamp: "Formatted at 2026-06-20 02:41:00 on dist-test-slave-rpg6"
server_key: "b4da8fa2c8c1a177d2dba607aac072e0"
server_key_iv: "085267f589b2580755f452d7d88e501e"
server_key_version: "encryptionkey@0"
I20260620 02:41:00.477025 20522 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:41:00.481326 20536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:00.483487 20522 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260620 02:41:00.483741 20522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/wal
uuid: "a55cf621b4284cfb93a39fc973efc87d"
format_stamp: "Formatted at 2026-06-20 02:41:00 on dist-test-slave-rpg6"
server_key: "b4da8fa2c8c1a177d2dba607aac072e0"
server_key_iv: "085267f589b2580755f452d7d88e501e"
server_key_version: "encryptionkey@0"
I20260620 02:41:00.484117 20522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:00.508982 20522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:00.510040 20522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:00.510421 20522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:00.512054 20522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:41:00.514240 20522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:41:00.514431 20522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:00.514582 20522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:41:00.514703 20522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:00.561796 20522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45075
I20260620 02:41:00.561939 20648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45075 every 8 connection(s)
I20260620 02:41:00.563339 20522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:41:00.572871 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 20522
I20260620 02:41:00.573100 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:41:00.573583 388 external_mini_cluster.cc:1428] Setting key 9ef0a588e2eb8b5df8f18c2d80ea58ca
I20260620 02:41:00.576830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39019
--builtin_ntp_servers=127.0.97.20:45199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:41:00.580765 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:00.566963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43865 (local address 127.0.97.62:39019)
0620 02:41:00.567502 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:00.567507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:00.569277 (+ 1770us) server_negotiation.cc:407] Connection header received
0620 02:41:00.571295 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:00.571301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:00.571407 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:00.571534 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:00.573064 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:00.574108 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:00.576411 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:00.576988 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:00.577126 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:00.579162 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:00.579169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:00.579172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:00.579199 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:00.579254 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:00.579270 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:00.579399 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:00.579706 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 02:41:00.580094 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":318,"threads_started":1}
I20260620 02:41:00.582147 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:00.567722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39019 (local address 127.0.97.2:43865)
0620 02:41:00.569062 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:00.569103 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:00.570986 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:00.571869 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:00.571886 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:00.571989 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:00.572768 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:00.572799 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:00.574298 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:00.574302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:00.576101 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:00.576119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:00.577245 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:00.577256 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:00.577602 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:00.578537 (+ 935us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:41:00.578541 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:00.578608 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:41:00.578756 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:00.579399 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:00.579415 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:00.579429 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 02:41:00.579681 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 02:41:00.580751 (+ 1070us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":354,"threads_started":1}
I20260620 02:41:00.584805 20649 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39019
I20260620 02:41:00.585260 20649 heartbeater.cc:464] Registering TS with master...
I20260620 02:41:00.586417 20649 heartbeater.cc:511] Master 127.0.97.62:39019 requested a full tablet report, sending...
I20260620 02:41:00.589781 20332 ts_manager.cc:194] Registered new tserver with Master: a55cf621b4284cfb93a39fc973efc87d (127.0.97.2:45075)
I20260620 02:41:00.591234 20332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:43865
I20260620 02:41:00.974258 20518 heartbeater.cc:503] Master 127.0.97.62:39019 was elected leader, sending a full tablet report...
W20260620 02:41:01.011951 20653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:01.012676 20653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:01.012822 20653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:01.021709 20653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:01.021989 20653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:01.022118 20653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:41:01.033521 20653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:01.034976 20653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:01.037158 20653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:01.060456 20661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:01.060711 20658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:01.060407 20659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:01.062668 20653 server_base.cc:1034] running on GCE node
I20260620 02:41:01.063699 20653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:01.065796 20653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:01.067139 20653 hybrid_clock.cc:648] HybridClock initialized: now 1781923261067068 us; error 118 us; skew 500 ppm
I20260620 02:41:01.067660 20653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:41:01.072206 20653 webserver.cc:460] Webserver started at http://127.0.97.3:34321/ using document root <none> and password file <none>
I20260620 02:41:01.073463 20653 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:01.073725 20653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:01.074189 20653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:01.078611 20653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/data/instance:
uuid: "d475553cadcb42f0bb771762d881128e"
format_stamp: "Formatted at 2026-06-20 02:41:01 on dist-test-slave-rpg6"
server_key: "d8a06bc1dbd9ec1ad09ac76b707371dd"
server_key_iv: "600ad52f0633e2e96b23e72f85f148eb"
server_key_version: "encryptionkey@0"
I20260620 02:41:01.079697 20653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d475553cadcb42f0bb771762d881128e"
format_stamp: "Formatted at 2026-06-20 02:41:01 on dist-test-slave-rpg6"
server_key: "d8a06bc1dbd9ec1ad09ac76b707371dd"
server_key_iv: "600ad52f0633e2e96b23e72f85f148eb"
server_key_version: "encryptionkey@0"
I20260620 02:41:01.089252 20653 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260620 02:41:01.096470 20667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:01.099231 20653 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260620 02:41:01.099519 20653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/wal
uuid: "d475553cadcb42f0bb771762d881128e"
format_stamp: "Formatted at 2026-06-20 02:41:01 on dist-test-slave-rpg6"
server_key: "d8a06bc1dbd9ec1ad09ac76b707371dd"
server_key_iv: "600ad52f0633e2e96b23e72f85f148eb"
server_key_version: "encryptionkey@0"
I20260620 02:41:01.100399 20653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:01.138078 20653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:01.139393 20653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:01.140058 20653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:01.141646 20653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:41:01.144528 20653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:41:01.144737 20653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:01.145020 20653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:41:01.145126 20653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:01.197301 20653 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43815
I20260620 02:41:01.197531 20779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43815 every 8 connection(s)
I20260620 02:41:01.198879 20653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:41:01.207041 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 20653
I20260620 02:41:01.207229 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:41:01.207795 388 external_mini_cluster.cc:1428] Setting key f28a41ebf1f3c630fab0ed415a595bf7
I20260620 02:41:01.216403 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.202469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35109 (local address 127.0.97.62:39019)
0620 02:41:01.203266 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:01.203275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:01.204830 (+ 1555us) server_negotiation.cc:407] Connection header received
0620 02:41:01.206895 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:01.206905 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:01.207008 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:01.207158 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:01.208697 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.209847 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.212723 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.213148 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.213301 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.215322 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:01.215329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:01.215332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:01.215356 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:01.215403 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:01.215418 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:01.215534 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:01.215799 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 02:41:01.215961 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":400,"threads_started":1}
I20260620 02:41:01.217764 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.203292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39019 (local address 127.0.97.3:35109)
0620 02:41:01.204599 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.204654 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:01.206481 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:01.207451 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:01.207468 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:01.207597 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:01.208474 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.208504 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.210023 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.210029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.212507 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.212527 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.213414 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.213423 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.213857 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.214631 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:41:01.214634 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:01.214682 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:41:01.214931 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:01.215534 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:01.215548 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:01.215558 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 02:41:01.215772 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 02:41:01.216344 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":481,"threads_started":1}
I20260620 02:41:01.220120 20780 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39019
I20260620 02:41:01.220615 20780 heartbeater.cc:464] Registering TS with master...
I20260620 02:41:01.221571 20780 heartbeater.cc:511] Master 127.0.97.62:39019 requested a full tablet report, sending...
I20260620 02:41:01.223555 20332 ts_manager.cc:194] Registered new tserver with Master: d475553cadcb42f0bb771762d881128e (127.0.97.3:43815)
I20260620 02:41:01.224385 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:41:01.224655 20332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:35109
I20260620 02:41:01.237906 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.229984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.0.97.62:39019)
0620 02:41:01.230318 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:01.230335 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:01.230603 (+ 268us) server_negotiation.cc:407] Connection header received
0620 02:41:01.230725 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:01.230730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:01.230821 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:01.230922 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:01.231508 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.232447 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.234514 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.235027 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.235164 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.236838 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:01.236846 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:01.236851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:01.236882 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:01.236955 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:01.236976 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:01.237107 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:01.237400 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 02:41:01.237563 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260620 02:41:01.245918 20332 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:41:01.253854 20332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:41:01.289103 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.273981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.0.97.3:43815)
0620 02:41:01.275106 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:01.275120 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:01.276702 (+ 1582us) server_negotiation.cc:407] Connection header received
0620 02:41:01.276830 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:01.276840 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:01.277155 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:01.277423 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:41:01.278864 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.280615 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.284260 (+ 3645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.286004 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.286159 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.286501 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:01.288155 (+ 1654us) server_negotiation.cc:299] Negotiation successful
0620 02:41:01.288464 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":620,"threads_started":1}
I20260620 02:41:01.289629 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.273397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43815 (local address 127.0.0.1:46056)
0620 02:41:01.276408 (+ 3011us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.276435 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:01.276563 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:01.277860 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:01.277873 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:01.277919 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:41:01.278452 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.278471 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.280801 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.280807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.283959 (+ 3152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.283990 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.286473 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.286484 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.287012 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.287255 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 02:41:01.287445 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 02:41:01.287673 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2885,"mutex_wait_us":8,"thread_start_us":2708,"threads_started":1}
I20260620 02:41:01.289719 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.276881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.0.97.1:32963)
0620 02:41:01.279093 (+ 2212us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:01.279099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:01.279123 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:41:01.279206 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:01.279243 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:01.279439 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:01.279624 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:41:01.280514 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.281761 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.285254 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.286780 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.286927 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.287780 (+ 853us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:01.288622 (+ 842us) server_negotiation.cc:299] Negotiation successful
0620 02:41:01.288896 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2044,"thread_start_us":1913,"threads_started":1}
I20260620 02:41:01.290266 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.273265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45075 (local address 127.0.0.1:46460)
0620 02:41:01.275438 (+ 2173us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.275484 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:01.275685 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:01.278930 (+ 3245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:01.278934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:01.278966 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:41:01.279381 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.279396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.281496 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.281501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.284435 (+ 2934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.284451 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.287375 (+ 2924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.287386 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.287943 (+ 557us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.287979 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:41:01.288067 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 02:41:01.288317 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1876,"mutex_wait_us":206,"thread_start_us":1711,"threads_started":1}
I20260620 02:41:01.290731 20797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.276625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32963 (local address 127.0.0.1:35452)
0620 02:41:01.277969 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.277992 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:01.278118 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:01.279888 (+ 1770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:01.279892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:01.279917 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:41:01.280310 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.280322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.281923 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.281934 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.285004 (+ 3070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.285021 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.287428 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.287450 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.288070 (+ 620us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.288121 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 02:41:01.288276 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 02:41:01.288467 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"spinlock_wait_cycles":7040,"thread_start_us":995,"threads_started":1}
I20260620 02:41:01.291127 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.273675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.0.97.2:45075)
0620 02:41:01.276435 (+ 2760us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:01.276446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:01.276482 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:41:01.278426 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:01.278433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:01.278697 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:01.279597 (+ 900us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:41:01.279853 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.281336 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.284708 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.286484 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.288403 (+ 1919us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.288690 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:01.288786 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 02:41:01.289858 (+ 1072us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2560,"thread_start_us":3488,"threads_started":1}
I20260620 02:41:01.295043 20584 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2e4a0d84894be4aa44adbcc0b2c30e (DEFAULT_TABLE table=test-table [id=eb7055ded7ac41dcb8d8ec33f97b6dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:41:01.295353 20453 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2e4a0d84894be4aa44adbcc0b2c30e (DEFAULT_TABLE table=test-table [id=eb7055ded7ac41dcb8d8ec33f97b6dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:41:01.295356 20715 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2e4a0d84894be4aa44adbcc0b2c30e (DEFAULT_TABLE table=test-table [id=eb7055ded7ac41dcb8d8ec33f97b6dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:41:01.296770 20584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2e4a0d84894be4aa44adbcc0b2c30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:01.296837 20453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2e4a0d84894be4aa44adbcc0b2c30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:01.296857 20715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2e4a0d84894be4aa44adbcc0b2c30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:01.312040 20799 tablet_bootstrap.cc:492] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d: Bootstrap starting.
I20260620 02:41:01.315768 20799 tablet_bootstrap.cc:654] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:01.316737 20800 tablet_bootstrap.cc:492] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: Bootstrap starting.
I20260620 02:41:01.318025 20799 log.cc:824] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:01.322721 20800 tablet_bootstrap.cc:654] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:01.325295 20800 log.cc:824] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:01.325726 20801 tablet_bootstrap.cc:492] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: Bootstrap starting.
I20260620 02:41:01.329216 20799 tablet_bootstrap.cc:492] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d: No bootstrap required, opened a new log
I20260620 02:41:01.329551 20799 ts_tablet_manager.cc:1397] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.003s
I20260620 02:41:01.330865 20800 tablet_bootstrap.cc:492] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: No bootstrap required, opened a new log
I20260620 02:41:01.331187 20800 ts_tablet_manager.cc:1397] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.002s
I20260620 02:41:01.331650 20801 tablet_bootstrap.cc:654] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:01.332845 20801 log.cc:824] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:01.335935 20801 tablet_bootstrap.cc:492] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: No bootstrap required, opened a new log
I20260620 02:41:01.336258 20801 ts_tablet_manager.cc:1397] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 02:41:01.343664 20799 raft_consensus.cc:348] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55cf621b4284cfb93a39fc973efc87d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45075 } } peers { permanent_uuid: "d475553cadcb42f0bb771762d881128e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43815 } } peers { permanent_uuid: "888b69e6bff54b5291af9f59da6bfa06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32963 } }
I20260620 02:41:01.344277 20799 raft_consensus.cc:374] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:01.344388 20799 raft_consensus.cc:729] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55cf621b4284cfb93a39fc973efc87d, State: Initialized, Role: FOLLOWER
I20260620 02:41:01.344871 20800 raft_consensus.cc:348] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55cf621b4284cfb93a39fc973efc87d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45075 } } peers { permanent_uuid: "d475553cadcb42f0bb771762d881128e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43815 } } peers { permanent_uuid: "888b69e6bff54b5291af9f59da6bfa06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32963 } }
I20260620 02:41:01.345350 20800 raft_consensus.cc:374] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:01.345453 20800 raft_consensus.cc:729] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888b69e6bff54b5291af9f59da6bfa06, State: Initialized, Role: FOLLOWER
I20260620 02:41:01.345263 20799 consensus_queue.cc:260] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55cf621b4284cfb93a39fc973efc87d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45075 } } peers { permanent_uuid: "d475553cadcb42f0bb771762d881128e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43815 } } peers { permanent_uuid: "888b69e6bff54b5291af9f59da6bfa06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32963 } }
I20260620 02:41:01.345160 20801 raft_consensus.cc:348] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55cf621b4284cfb93a39fc973efc87d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45075 } } peers { permanent_uuid: "d475553cadcb42f0bb771762d881128e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43815 } } peers { permanent_uuid: "888b69e6bff54b5291af9f59da6bfa06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32963 } }
I20260620 02:41:01.345649 20801 raft_consensus.cc:374] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:01.346095 20801 raft_consensus.cc:729] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d475553cadcb42f0bb771762d881128e, State: Initialized, Role: FOLLOWER
I20260620 02:41:01.346264 20800 consensus_queue.cc:260] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55cf621b4284cfb93a39fc973efc87d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45075 } } peers { permanent_uuid: "d475553cadcb42f0bb771762d881128e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43815 } } peers { permanent_uuid: "888b69e6bff54b5291af9f59da6bfa06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32963 } }
I20260620 02:41:01.347137 20801 consensus_queue.cc:260] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55cf621b4284cfb93a39fc973efc87d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45075 } } peers { permanent_uuid: "d475553cadcb42f0bb771762d881128e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43815 } } peers { permanent_uuid: "888b69e6bff54b5291af9f59da6bfa06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32963 } }
I20260620 02:41:01.347513 20649 heartbeater.cc:503] Master 127.0.97.62:39019 was elected leader, sending a full tablet report...
I20260620 02:41:01.349033 20800 ts_tablet_manager.cc:1428] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: Time spent starting tablet: real 0.018s user 0.017s sys 0.002s
I20260620 02:41:01.351399 20799 ts_tablet_manager.cc:1428] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d: Time spent starting tablet: real 0.022s user 0.015s sys 0.004s
I20260620 02:41:01.353229 20780 heartbeater.cc:503] Master 127.0.97.62:39019 was elected leader, sending a full tablet report...
I20260620 02:41:01.356078 20801 ts_tablet_manager.cc:1428] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: Time spent starting tablet: real 0.020s user 0.010s sys 0.008s
I20260620 02:41:01.377808 20807 raft_consensus.cc:482] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:41:01.378361 20807 raft_consensus.cc:504] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55cf621b4284cfb93a39fc973efc87d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45075 } } peers { permanent_uuid: "d475553cadcb42f0bb771762d881128e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43815 } } peers { permanent_uuid: "888b69e6bff54b5291af9f59da6bfa06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32963 } }
I20260620 02:41:01.381565 20807 leader_election.cc:288] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a55cf621b4284cfb93a39fc973efc87d (127.0.97.2:45075), 888b69e6bff54b5291af9f59da6bfa06 (127.0.97.1:32963)
I20260620 02:41:01.389600 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.382287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34867 (local address 127.0.97.1:32963)
0620 02:41:01.382524 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:01.382530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:01.382556 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:41:01.382658 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:01.382668 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:01.382805 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:01.382933 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:41:01.384258 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.385330 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.387593 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.388527 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.388700 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.388835 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:01.389129 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 02:41:01.389274 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:41:01.389639 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.381887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32963 (local address 127.0.97.3:34867)
0620 02:41:01.382160 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.382183 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:01.382337 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:01.383196 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:01.383206 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:01.383250 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:41:01.383705 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.383735 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.385631 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.385640 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.387328 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.387346 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.388739 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.388745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.388944 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.388971 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 02:41:01.389096 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 02:41:01.389274 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":1}
I20260620 02:41:01.390461 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.382854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51867 (local address 127.0.97.2:45075)
0620 02:41:01.383042 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:01.383050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:01.383870 (+ 820us) server_negotiation.cc:407] Connection header received
0620 02:41:01.384092 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:01.384096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:01.384180 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:01.384293 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:41:01.384890 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.385595 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.388361 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.389309 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.389460 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.389599 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:01.390001 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 02:41:01.390187 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:41:01.391211 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.382337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45075 (local address 127.0.97.3:51867)
0620 02:41:01.383717 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.383733 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:01.383924 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:01.384312 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:01.384316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:01.384341 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:41:01.384708 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.384718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.385780 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.385786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.387757 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:01.387772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.389507 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.389513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:01.389763 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.389814 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 02:41:01.390668 (+ 854us) client_negotiation.cc:240] Negotiation successful
0620 02:41:01.390850 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"mutex_wait_us":91,"thread_start_us":879,"threads_started":1}
I20260620 02:41:01.391911 20473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2e4a0d84894be4aa44adbcc0b2c30e" candidate_uuid: "d475553cadcb42f0bb771762d881128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "888b69e6bff54b5291af9f59da6bfa06" is_pre_election: true
I20260620 02:41:01.392594 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2e4a0d84894be4aa44adbcc0b2c30e" candidate_uuid: "d475553cadcb42f0bb771762d881128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55cf621b4284cfb93a39fc973efc87d" is_pre_election: true
I20260620 02:41:01.392887 20473 raft_consensus.cc:2454] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d475553cadcb42f0bb771762d881128e in term 0.
I20260620 02:41:01.393292 20604 raft_consensus.cc:2454] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d475553cadcb42f0bb771762d881128e in term 0.
I20260620 02:41:01.394320 20669 leader_election.cc:302] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888b69e6bff54b5291af9f59da6bfa06, d475553cadcb42f0bb771762d881128e; no voters:
I20260620 02:41:01.394909 20807 raft_consensus.cc:2781] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:41:01.395179 20807 raft_consensus.cc:482] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:41:01.395290 20807 raft_consensus.cc:3037] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:01.397918 20807 raft_consensus.cc:504] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55cf621b4284cfb93a39fc973efc87d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45075 } } peers { permanent_uuid: "d475553cadcb42f0bb771762d881128e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43815 } } peers { permanent_uuid: "888b69e6bff54b5291af9f59da6bfa06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32963 } }
I20260620 02:41:01.398672 20807 leader_election.cc:288] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [CANDIDATE]: Term 1 election: Requested vote from peers a55cf621b4284cfb93a39fc973efc87d (127.0.97.2:45075), 888b69e6bff54b5291af9f59da6bfa06 (127.0.97.1:32963)
I20260620 02:41:01.399477 20604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2e4a0d84894be4aa44adbcc0b2c30e" candidate_uuid: "d475553cadcb42f0bb771762d881128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55cf621b4284cfb93a39fc973efc87d"
I20260620 02:41:01.399775 20604 raft_consensus.cc:3037] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:01.399749 20473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2e4a0d84894be4aa44adbcc0b2c30e" candidate_uuid: "d475553cadcb42f0bb771762d881128e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "888b69e6bff54b5291af9f59da6bfa06"
I20260620 02:41:01.400049 20473 raft_consensus.cc:3037] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:01.403811 20473 raft_consensus.cc:2454] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d475553cadcb42f0bb771762d881128e in term 1.
I20260620 02:41:01.403802 20604 raft_consensus.cc:2454] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d475553cadcb42f0bb771762d881128e in term 1.
I20260620 02:41:01.404541 20671 leader_election.cc:302] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a55cf621b4284cfb93a39fc973efc87d, d475553cadcb42f0bb771762d881128e; no voters:
I20260620 02:41:01.405043 20807 raft_consensus.cc:2781] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:01.405586 20807 raft_consensus.cc:686] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 1 LEADER]: Becoming Leader. State: Replica: d475553cadcb42f0bb771762d881128e, State: Running, Role: LEADER
I20260620 02:41:01.406176 20807 consensus_queue.cc:237] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55cf621b4284cfb93a39fc973efc87d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45075 } } peers { permanent_uuid: "d475553cadcb42f0bb771762d881128e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43815 } } peers { permanent_uuid: "888b69e6bff54b5291af9f59da6bfa06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32963 } }
I20260620 02:41:01.413286 20330 catalog_manager.cc:5495] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e reported cstate change: term changed from 0 to 1, leader changed from <none> to d475553cadcb42f0bb771762d881128e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d475553cadcb42f0bb771762d881128e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55cf621b4284cfb93a39fc973efc87d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d475553cadcb42f0bb771762d881128e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888b69e6bff54b5291af9f59da6bfa06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32963 } health_report { overall_health: UNKNOWN } } }
W20260620 02:41:01.442636 20519 tablet.cc:2369] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:41:01.451279 20781 tablet.cc:2369] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:41:01.459785 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.451229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46068 (local address 127.0.97.3:43815)
0620 02:41:01.451730 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:01.451739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:01.451774 (+ 35us) server_negotiation.cc:407] Connection header received
0620 02:41:01.451984 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:01.451991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:01.452133 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:01.452372 (+ 239us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:41:01.452958 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.453967 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.455915 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.456620 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.456811 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.457934 (+ 1123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:41:01.458893 (+ 959us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:41:01.459103 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:01.459436 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 02:41:01.459544 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 02:41:01.472728 20604 raft_consensus.cc:1261] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d [term 1 FOLLOWER]: Refusing update from remote peer d475553cadcb42f0bb771762d881128e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:41:01.472911 20473 raft_consensus.cc:1261] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06 [term 1 FOLLOWER]: Refusing update from remote peer d475553cadcb42f0bb771762d881128e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:41:01.474429 20809 consensus_queue.cc:1035] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [LEADER]: Connected to new peer: Peer: permanent_uuid: "888b69e6bff54b5291af9f59da6bfa06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:41:01.474862 20807 consensus_queue.cc:1035] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55cf621b4284cfb93a39fc973efc87d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:41:01.515466 20330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:41:01.515817 20330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:41:01.521018 20330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f74632f5f3db44fe9c98a5205418d57a: Sending DeleteTablet for 3 replicas of tablet 7d2e4a0d84894be4aa44adbcc0b2c30e
I20260620 02:41:01.523123 20715 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2e4a0d84894be4aa44adbcc0b2c30e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:41:01 UTC) from {username='slave'} at 127.0.0.1:46056
I20260620 02:41:01.523116 20584 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2e4a0d84894be4aa44adbcc0b2c30e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:41:01 UTC) from {username='slave'} at 127.0.0.1:46460
I20260620 02:41:01.524089 20453 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2e4a0d84894be4aa44adbcc0b2c30e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:41:01 UTC) from {username='slave'} at 127.0.0.1:35452
I20260620 02:41:01.524355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 20391
I20260620 02:41:01.524561 20822 tablet_replica.cc:330] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: stopping tablet replica
I20260620 02:41:01.525100 20822 raft_consensus.cc:2229] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:41:01.525876 20822 raft_consensus.cc:2258] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:41:01.526753 20823 tablet_replica.cc:330] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d: stopping tablet replica
I20260620 02:41:01.527195 20823 raft_consensus.cc:2229] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:41:01.527908 20823 raft_consensus.cc:2258] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:41:01.530627 20822 ts_tablet_manager.cc:1905] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:41:01.534323 20823 ts_tablet_manager.cc:1905] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:41:01.537612 20824 tablet_replica.cc:330] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: stopping tablet replica
I20260620 02:41:01.538383 20824 raft_consensus.cc:2229] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:41:01.539037 20824 raft_consensus.cc:2258] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:41:01.544075 20822 ts_tablet_manager.cc:1918] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:41:01.544349 20822 log.cc:1196] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-2/wal/wals/7d2e4a0d84894be4aa44adbcc0b2c30e
I20260620 02:41:01.544402 20824 ts_tablet_manager.cc:1905] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:41:01.545506 20822 ts_tablet_manager.cc:1939] T 7d2e4a0d84894be4aa44adbcc0b2c30e P d475553cadcb42f0bb771762d881128e: Deleting consensus metadata
I20260620 02:41:01.547188 20823 ts_tablet_manager.cc:1918] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:41:01.547667 20823 log.cc:1196] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-1/wal/wals/7d2e4a0d84894be4aa44adbcc0b2c30e
I20260620 02:41:01.548048 20317 catalog_manager.cc:4841] TS d475553cadcb42f0bb771762d881128e (127.0.97.3:43815): tablet 7d2e4a0d84894be4aa44adbcc0b2c30e (table test-table [id=eb7055ded7ac41dcb8d8ec33f97b6dc1]) successfully deleted
I20260620 02:41:01.548866 20823 ts_tablet_manager.cc:1939] T 7d2e4a0d84894be4aa44adbcc0b2c30e P a55cf621b4284cfb93a39fc973efc87d: Deleting consensus metadata
I20260620 02:41:01.552274 20319 catalog_manager.cc:4841] TS a55cf621b4284cfb93a39fc973efc87d (127.0.97.2:45075): tablet 7d2e4a0d84894be4aa44adbcc0b2c30e (table test-table [id=eb7055ded7ac41dcb8d8ec33f97b6dc1]) successfully deleted
I20260620 02:41:01.556481 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.526297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.0.97.1:32963)
0620 02:41:01.526694 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:01.526701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:01.536462 (+ 9761us) server_negotiation.cc:407] Connection header received
0620 02:41:01.536754 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:01.536886 (+ 132us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:01.537234 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:01.537601 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:01.538659 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.540673 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.550773 (+ 10100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.551741 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.551937 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.553882 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:01.553899 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:01.553937 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:01.555329 (+ 1392us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:01.555404 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:01.555435 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:01.555576 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:01.555892 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 02:41:01.556047 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260620 02:41:01.556473 20824 ts_tablet_manager.cc:1918] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:41:01.556915 20824 log.cc:1196] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: Deleting WAL directory at /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781922915737669-388-0/minicluster-data/ts-0/wal/wals/7d2e4a0d84894be4aa44adbcc0b2c30e
I20260620 02:41:01.557638 20824 ts_tablet_manager.cc:1939] T 7d2e4a0d84894be4aa44adbcc0b2c30e P 888b69e6bff54b5291af9f59da6bfa06: Deleting consensus metadata
I20260620 02:41:01.557790 20513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:01.696183 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 20391
W20260620 02:41:01.726581 20317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:32963 (error 108)
I20260620 02:41:01.728142 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 20522
I20260620 02:41:01.729760 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.729259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32963 (local address 127.0.0.1:35462)
0620 02:41:01.729435 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.729568 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:41:01.730084 20317 catalog_manager.cc:4584] TS 888b69e6bff54b5291af9f59da6bfa06 (127.0.97.1:32963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d2e4a0d84894be4aa44adbcc0b2c30e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32963: connect: Connection refused (error 111)
I20260620 02:41:01.736796 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.729056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.0.97.2:45075)
0620 02:41:01.729269 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:01.729276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:01.729490 (+ 214us) server_negotiation.cc:407] Connection header received
0620 02:41:01.729870 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:01.729878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:01.730007 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:01.730256 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:01.730671 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.731594 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.733101 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.733550 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.733711 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.735188 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:01.735199 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:01.735215 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:01.735933 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:01.735990 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:01.736020 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:01.736168 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:01.736297 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 02:41:01.736418 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:41:01.737591 20644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:01.784554 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.783918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32963 (local address 127.0.0.1:35472)
0620 02:41:01.784184 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.784326 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 02:41:01.786633 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.785841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32963 (local address 127.0.0.1:35478)
0620 02:41:01.786295 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.786434 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 02:41:01.848166 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.846922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32963 (local address 127.0.0.1:35488)
0620 02:41:01.847515 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.847717 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260620 02:41:01.850257 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.849701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32963 (local address 127.0.0.1:35500)
0620 02:41:01.849925 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.850039 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 02:41:01.867552 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 20522
I20260620 02:41:01.905654 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 20653
I20260620 02:41:01.918751 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.907046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.0.97.3:43815)
0620 02:41:01.907434 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:01.907440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:01.907463 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:41:01.908477 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:01.908506 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:01.908681 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:01.908853 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:01.910073 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.911128 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.913337 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:01.913840 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:01.913975 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:01.916309 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:01.916324 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:01.916340 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:01.917381 (+ 1041us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:01.917445 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:01.917473 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:01.917582 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:01.918156 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 02:41:01.918334 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260620 02:41:01.919669 20775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:01.933380 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.931743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32963 (local address 127.0.0.1:35508)
0620 02:41:01.932233 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.932382 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
I20260620 02:41:01.937917 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:01.934986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32963 (local address 127.0.0.1:35516)
0620 02:41:01.936143 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:01.937579 (+ 1436us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":885}
I20260620 02:41:02.099130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 20653
I20260620 02:41:02.113381 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:02.112444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32963 (local address 127.0.0.1:35528)
0620 02:41:02.112885 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:02.113058 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260620 02:41:02.115963 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:02.114584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32963 (local address 127.0.0.1:35538)
0620 02:41:02.115143 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:02.115573 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 02:41:02.130152 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 20301
I20260620 02:41:02.131467 20362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:02.278944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 20301
2026-06-20T02:41:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3762 ms)
[----------] 6 tests from EncryptionPolicyTest (28849 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:02 dist-test-slave-rpg6 krb5kdc[20833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:02 dist-test-slave-rpg6 krb5kdc[20833](info): set up 2 sockets
Jun 20 02:41:02 dist-test-slave-rpg6 krb5kdc[20833](info): commencing operation
krb5kdc: starting...
W20260620 02:41:03.597374 388 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:03 dist-test-slave-rpg6 krb5kdc[20833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:03.957332 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:04.396221 20846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:04.396729 20846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:04.396944 20846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:04.406891 20846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:04.407074 20846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:04.407202 20846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:04.407303 20846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:04.407431 20846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:04.419411 20846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:04.421013 20846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:04.422562 20846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:04.440137 20854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:04.440068 20852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:04.440150 20851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:04.441077 20846 server_base.cc:1034] running on GCE node
I20260620 02:41:04.441973 20846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:04.443670 20846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:04.445217 20846 hybrid_clock.cc:648] HybridClock initialized: now 1781923264445096 us; error 52 us; skew 500 ppm
I20260620 02:41:04.445963 20846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:04 dist-test-slave-rpg6 krb5kdc[20833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:04.450758 20846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:04.453621 20846 webserver.cc:460] Webserver started at http://127.0.0.1:37243/ using document root <none> and password file <none>
I20260620 02:41:04.454571 20846 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:04.454782 20846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:04.455166 20846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:04.459287 20846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "481dff121fc84dbea3711c322ffea7e9"
format_stamp: "Formatted at 2026-06-20 02:41:04 on dist-test-slave-rpg6"
server_key: "3b678eae70202ecc0105252101d495e5"
server_key_iv: "117ca01cb27ea6ddf9625a4dace7f6a8"
server_key_version: "encryptionkey@0"
I20260620 02:41:04.460399 20846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "481dff121fc84dbea3711c322ffea7e9"
format_stamp: "Formatted at 2026-06-20 02:41:04 on dist-test-slave-rpg6"
server_key: "3b678eae70202ecc0105252101d495e5"
server_key_iv: "117ca01cb27ea6ddf9625a4dace7f6a8"
server_key_version: "encryptionkey@0"
I20260620 02:41:04.467680 20846 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 02:41:04.474200 20861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:04.476714 20846 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:41:04.477030 20846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "481dff121fc84dbea3711c322ffea7e9"
format_stamp: "Formatted at 2026-06-20 02:41:04 on dist-test-slave-rpg6"
server_key: "3b678eae70202ecc0105252101d495e5"
server_key_iv: "117ca01cb27ea6ddf9625a4dace7f6a8"
server_key_version: "encryptionkey@0"
I20260620 02:41:04.477293 20846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:04.520267 20846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:04.527143 20846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:04.527814 20846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:04.554713 20846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36189
I20260620 02:41:04.554749 20912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36189 every 8 connection(s)
I20260620 02:41:04.556461 20846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:04.561383 20913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:04.563803 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 20846
I20260620 02:41:04.564157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:04.564850 388 external_mini_cluster.cc:1428] Setting key 114da4845a0a04e62b2f0f0b2bfebfcf
I20260620 02:41:04.577437 20913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9: Bootstrap starting.
I20260620 02:41:04.582096 20913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:04.583585 20913 log.cc:824] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:04.587327 20913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9: No bootstrap required, opened a new log
Jun 20 02:41:04 dist-test-slave-rpg6 krb5kdc[20833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:41:04.597750 20913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dff121fc84dbea3711c322ffea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } }
I20260620 02:41:04.598254 20913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:04.598448 20913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481dff121fc84dbea3711c322ffea7e9, State: Initialized, Role: FOLLOWER
I20260620 02:41:04.599550 20913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dff121fc84dbea3711c322ffea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } }
I20260620 02:41:04.600039 20913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:04.600220 20913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:04.600466 20913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:04.603195 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:04.567469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50251 (local address 127.0.0.1:36189)
0620 02:41:04.568404 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:04.568423 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:04.568479 (+ 56us) server_negotiation.cc:407] Connection header received
0620 02:41:04.571450 (+ 2971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:04.571489 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:04.572393 (+ 904us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:04.572985 (+ 592us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:04.575466 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:04.578672 (+ 3206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:04.581200 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:04.583153 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:04.583518 (+ 365us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:04.589726 (+ 6208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:04.589746 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:04.589773 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:04.590077 (+ 304us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:04.594856 (+ 4779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:04.597183 (+ 2327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:04.597196 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:04.597207 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:04.597522 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:04.598217 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:04.598223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:04.598226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:04.599537 (+ 1311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:04.600103 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:04.600982 (+ 879us) server_negotiation.cc:299] Negotiation successful
0620 02:41:04.601511 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":395,"threads_started":1}
I20260620 02:41:04.604195 20913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dff121fc84dbea3711c322ffea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } }
I20260620 02:41:04.604770 20913 leader_election.cc:302] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 481dff121fc84dbea3711c322ffea7e9; no voters:
I20260620 02:41:04.605412 20913 leader_election.cc:288] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:04.605609 20918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:04.608497 20913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:04.609887 20918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [term 1 LEADER]: Becoming Leader. State: Replica: 481dff121fc84dbea3711c322ffea7e9, State: Running, Role: LEADER
I20260620 02:41:04.610287 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:04.611136 20918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dff121fc84dbea3711c322ffea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } }
I20260620 02:41:04.623359 20919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "481dff121fc84dbea3711c322ffea7e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dff121fc84dbea3711c322ffea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } } }
I20260620 02:41:04.626770 20919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:04.624699 20920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 481dff121fc84dbea3711c322ffea7e9. Latest consensus state: current_term: 1 leader_uuid: "481dff121fc84dbea3711c322ffea7e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dff121fc84dbea3711c322ffea7e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36189 } } }
I20260620 02:41:04.627416 20920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:04.633298 20933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:04.641208 20933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:04.660648 20933 catalog_manager.cc:1347] Generated new cluster ID: 75ba051d196a43fe8eb3839e1255f5c7
I20260620 02:41:04.660991 20933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:04.664865 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:04.636593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48713 (local address 127.0.0.1:36189)
0620 02:41:04.636858 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:04.636865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:04.637612 (+ 747us) server_negotiation.cc:407] Connection header received
0620 02:41:04.637917 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:04.637926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:04.638066 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:04.638235 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:04.641277 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:04.642560 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:04.645016 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:04.646267 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:04.646438 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:04.651811 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:04.651821 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:04.651889 (+ 68us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:04.651958 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:04.658757 (+ 6799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:04.661112 (+ 2355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:04.661122 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:04.661126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:04.661457 (+ 331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:04.662420 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:04.662430 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:04.662435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:04.663166 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:04.663635 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:04.664126 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 02:41:04.664357 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 02:41:04.684415 20933 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:04.686492 20933 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:04.701753 20933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 481dff121fc84dbea3711c322ffea7e9: Generated new TSK 0
I20260620 02:41:04.702986 20933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:04.718370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 20846
I20260620 02:41:04.720109 20908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:04.829843 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 20846
W20260620 02:41:04.850425 20831 connection.cc:535] client connection to 127.0.0.1:36189 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36189): Connection reset by peer (error 104)
2026-06-20T02:41:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2532 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:04 dist-test-slave-rpg6 krb5kdc[20946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:04 dist-test-slave-rpg6 krb5kdc[20946](info): set up 2 sockets
Jun 20 02:41:04 dist-test-slave-rpg6 krb5kdc[20946](info): commencing operation
krb5kdc: starting...
W20260620 02:41:06.052745 388 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.001s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:06 dist-test-slave-rpg6 krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:06.384646 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:06.805330 20959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:06.805811 20959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:06.805928 20959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:06.816699 20959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:06.816902 20959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:06.816964 20959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:06.817037 20959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:06.817097 20959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:06.831686 20959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:06.833686 20959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:06.835727 20959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:06.851092 20964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:06.851030 20965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:06.851432 20967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:06.852793 20959 server_base.cc:1034] running on GCE node
I20260620 02:41:06.854290 20959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:06.856771 20959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:06.858199 20959 hybrid_clock.cc:648] HybridClock initialized: now 1781923266858097 us; error 111 us; skew 500 ppm
I20260620 02:41:06.858726 20959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:06 dist-test-slave-rpg6 krb5kdc[20946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:06.867585 20959 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:06.870021 20959 webserver.cc:460] Webserver started at http://127.0.0.1:40891/ using document root <none> and password file <none>
I20260620 02:41:06.870949 20959 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:06.871091 20959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:06.871469 20959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:06.875349 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "cd443ba79ef9417fba70d5e934e37af3"
format_stamp: "Formatted at 2026-06-20 02:41:06 on dist-test-slave-rpg6"
server_key: "9b98fdf12acecb13fb668aa79bd578af"
server_key_iv: "dc10be710292c353f79fa788e807e5fd"
server_key_version: "encryptionkey@0"
I20260620 02:41:06.876502 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "cd443ba79ef9417fba70d5e934e37af3"
format_stamp: "Formatted at 2026-06-20 02:41:06 on dist-test-slave-rpg6"
server_key: "9b98fdf12acecb13fb668aa79bd578af"
server_key_iv: "dc10be710292c353f79fa788e807e5fd"
server_key_version: "encryptionkey@0"
I20260620 02:41:06.883240 20959 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:41:06.888543 20974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:06.891089 20959 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260620 02:41:06.891624 20959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "cd443ba79ef9417fba70d5e934e37af3"
format_stamp: "Formatted at 2026-06-20 02:41:06 on dist-test-slave-rpg6"
server_key: "9b98fdf12acecb13fb668aa79bd578af"
server_key_iv: "dc10be710292c353f79fa788e807e5fd"
server_key_version: "encryptionkey@0"
I20260620 02:41:06.892272 20959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:06.923707 20959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:06.929196 20959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:06.929554 20959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:06.959264 20959 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44815
I20260620 02:41:06.959280 21025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44815 every 8 connection(s)
I20260620 02:41:06.961258 20959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:06.965969 21026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:06.970095 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 20959
I20260620 02:41:06.970392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:06.971299 388 external_mini_cluster.cc:1428] Setting key b1b2d7db00e4e139d14ca08db1ff5285
I20260620 02:41:06.982440 21026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3: Bootstrap starting.
I20260620 02:41:06.987761 21026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:06.989607 21026 log.cc:824] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:06.994387 21026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3: No bootstrap required, opened a new log
I20260620 02:41:06.995528 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:06.974953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60295 (local address 127.0.0.1:44815)
0620 02:41:06.976291 (+ 1338us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:06.976312 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:06.976895 (+ 583us) server_negotiation.cc:407] Connection header received
0620 02:41:06.979966 (+ 3071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:06.980009 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:06.980723 (+ 714us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:06.981276 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:06.982272 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:06.984938 (+ 2666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:06.987496 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:06.988876 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:06.989171 (+ 295us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:06.991868 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:06.991886 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:06.991918 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:06.992841 (+ 923us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:06.992979 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:06.993022 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:06.993231 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:06.993764 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 02:41:06.993990 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":599,"threads_started":1}
I20260620 02:41:07.002213 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:07.005187 21026 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd443ba79ef9417fba70d5e934e37af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260620 02:41:07.005709 21026 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:07.005854 21026 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd443ba79ef9417fba70d5e934e37af3, State: Initialized, Role: FOLLOWER
I20260620 02:41:07.006563 21026 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd443ba79ef9417fba70d5e934e37af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260620 02:41:07.006875 21026 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:07.007056 21026 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:07.007314 21026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:07.010891 21026 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd443ba79ef9417fba70d5e934e37af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260620 02:41:07.011545 21026 leader_election.cc:302] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd443ba79ef9417fba70d5e934e37af3; no voters:
I20260620 02:41:07.012480 21026 leader_election.cc:288] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:07.012641 21036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:07.014505 21036 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [term 1 LEADER]: Becoming Leader. State: Replica: cd443ba79ef9417fba70d5e934e37af3, State: Running, Role: LEADER
I20260620 02:41:07.015630 21026 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:07.015182 21036 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd443ba79ef9417fba70d5e934e37af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260620 02:41:07.016187 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:07.007330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48099 (local address 127.0.0.1:44815)
0620 02:41:07.007599 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:07.007606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:07.008010 (+ 404us) server_negotiation.cc:407] Connection header received
0620 02:41:07.008185 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:07.008190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:07.008293 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:07.008446 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:07.009089 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:07.010311 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:07.012210 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:07.012955 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:07.013982 (+ 1027us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:07.014931 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:07.014936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:07.014939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:07.014963 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:07.015046 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:07.015064 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:07.015220 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:07.015435 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 02:41:07.015515 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260620 02:41:07.020784 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd443ba79ef9417fba70d5e934e37af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd443ba79ef9417fba70d5e934e37af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } } }
I20260620 02:41:07.020805 21038 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd443ba79ef9417fba70d5e934e37af3. Latest consensus state: current_term: 1 leader_uuid: "cd443ba79ef9417fba70d5e934e37af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd443ba79ef9417fba70d5e934e37af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } } }
I20260620 02:41:07.021498 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:07.021739 21038 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:07.024785 21043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:07.035091 21043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:07.051245 21043 catalog_manager.cc:1347] Generated new cluster ID: 83ebf8e7030a400a9a25d7b2fbe6164c
I20260620 02:41:07.051416 21043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:07.068073 21043 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:07.069871 21043 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:07.082832 21043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd443ba79ef9417fba70d5e934e37af3: Generated new TSK 0
I20260620 02:41:07.083981 21043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:07.100787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 20959
W20260620 02:41:07.102914 21021 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60295
W20260620 02:41:07.104457 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 20959: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:07.104743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 20959
W20260620 02:41:07.125489 20944 connection.cc:535] client connection to 127.0.0.1:44815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44815): Connection reset by peer (error 104)
2026-06-20T02:41:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:07 dist-test-slave-rpg6 krb5kdc[21057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:07 dist-test-slave-rpg6 krb5kdc[21057](info): set up 2 sockets
Jun 20 02:41:07 dist-test-slave-rpg6 krb5kdc[21057](info): commencing operation
krb5kdc: starting...
W20260620 02:41:08.336879 388 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:08 dist-test-slave-rpg6 krb5kdc[21057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:08.680150 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:09.093612 21070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:09.093981 21070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:09.094137 21070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:09.103408 21070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:09.103574 21070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:09.103623 21070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:09.104018 21070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:09.104681 21070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:09.124567 21070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:09.126169 21070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:09.127722 21070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:09.149502 21076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:09.150360 21075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:09.151252 21078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:09.152108 21070 server_base.cc:1034] running on GCE node
I20260620 02:41:09.153733 21070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:09.156054 21070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:09.157752 21070 hybrid_clock.cc:648] HybridClock initialized: now 1781923269157604 us; error 75 us; skew 500 ppm
I20260620 02:41:09.159124 21070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:09 dist-test-slave-rpg6 krb5kdc[21057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:09.163970 21070 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:09.166548 21070 webserver.cc:460] Webserver started at http://127.0.0.1:46249/ using document root <none> and password file <none>
I20260620 02:41:09.167680 21070 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:09.167928 21070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:09.168501 21070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:09.172613 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "78bcdbbb7474480ca980e5a361567093"
format_stamp: "Formatted at 2026-06-20 02:41:09 on dist-test-slave-rpg6"
server_key: "b605ae3080f9a95d42732903f8e94dc9"
server_key_iv: "903d0c66063b2b75b44ecfbdfa5cc026"
server_key_version: "encryptionkey@0"
I20260620 02:41:09.173617 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "78bcdbbb7474480ca980e5a361567093"
format_stamp: "Formatted at 2026-06-20 02:41:09 on dist-test-slave-rpg6"
server_key: "b605ae3080f9a95d42732903f8e94dc9"
server_key_iv: "903d0c66063b2b75b44ecfbdfa5cc026"
server_key_version: "encryptionkey@0"
I20260620 02:41:09.183399 21070 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260620 02:41:09.190013 21085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:09.192656 21070 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260620 02:41:09.192967 21070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "78bcdbbb7474480ca980e5a361567093"
format_stamp: "Formatted at 2026-06-20 02:41:09 on dist-test-slave-rpg6"
server_key: "b605ae3080f9a95d42732903f8e94dc9"
server_key_iv: "903d0c66063b2b75b44ecfbdfa5cc026"
server_key_version: "encryptionkey@0"
I20260620 02:41:09.193234 21070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:09.201392 21070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:09.209484 21070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:09.209959 21070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:09.242561 21070 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39205
I20260620 02:41:09.242822 21136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39205 every 8 connection(s)
I20260620 02:41:09.244596 21070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:09.246101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 21070
I20260620 02:41:09.246577 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:09.247161 388 external_mini_cluster.cc:1428] Setting key 9c2f841aaad3837768590329d2c367e3
I20260620 02:41:09.251328 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:09.258900 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:09.249728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42523 (local address 127.0.0.1:39205)
0620 02:41:09.251031 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:09.251055 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:09.251127 (+ 72us) server_negotiation.cc:407] Connection header received
0620 02:41:09.253600 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:09.253636 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:09.254090 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:09.254526 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:09.256199 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:09.256212 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:09.256244 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:09.256279 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:09.256450 (+ 171us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:09.256486 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:09.256850 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:09.257026 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 02:41:09.257419 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":456,"threads_started":1}
I20260620 02:41:09.266004 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:09.266430 21137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093: Bootstrap starting.
I20260620 02:41:09.271889 21137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:09.273643 21137 log.cc:824] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:09.280167 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:09.273553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57151 (local address 127.0.0.1:39205)
0620 02:41:09.273871 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:09.273881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:09.275301 (+ 1420us) server_negotiation.cc:407] Connection header received
0620 02:41:09.276114 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:09.276122 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:09.276228 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:09.276754 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:09.278199 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:09.278214 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:09.278225 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:09.278385 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:09.278459 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:09.278491 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:09.278793 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:09.278877 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 02:41:09.278959 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 02:41:09.282948 21137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093: No bootstrap required, opened a new log
I20260620 02:41:09.292439 21137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bcdbbb7474480ca980e5a361567093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39205 } }
I20260620 02:41:09.292972 21137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:09.293045 21137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bcdbbb7474480ca980e5a361567093, State: Initialized, Role: FOLLOWER
I20260620 02:41:09.293774 21137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bcdbbb7474480ca980e5a361567093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39205 } }
I20260620 02:41:09.294061 21137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:09.294231 21137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:09.294420 21137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:09.297283 21137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bcdbbb7474480ca980e5a361567093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39205 } }
I20260620 02:41:09.298000 21137 leader_election.cc:302] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78bcdbbb7474480ca980e5a361567093; no voters:
I20260620 02:41:09.299036 21137 leader_election.cc:288] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:09.299479 21147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:09.300741 21147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [term 1 LEADER]: Becoming Leader. State: Replica: 78bcdbbb7474480ca980e5a361567093, State: Running, Role: LEADER
I20260620 02:41:09.302380 21147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bcdbbb7474480ca980e5a361567093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39205 } }
I20260620 02:41:09.303470 21137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:09.306810 21148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78bcdbbb7474480ca980e5a361567093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bcdbbb7474480ca980e5a361567093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39205 } } }
I20260620 02:41:09.306839 21149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78bcdbbb7474480ca980e5a361567093. Latest consensus state: current_term: 1 leader_uuid: "78bcdbbb7474480ca980e5a361567093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bcdbbb7474480ca980e5a361567093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39205 } } }
I20260620 02:41:09.307566 21148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:09.308135 21149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:09.312534 21152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:09.319939 21152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:09.334892 21152 catalog_manager.cc:1347] Generated new cluster ID: c84926a2f71e4dbb93cab787acd7d23f
I20260620 02:41:09.335417 21152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:09.356436 21152 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:09.357038 21152 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:09.365167 21152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78bcdbbb7474480ca980e5a361567093: Generated new TSK 0
I20260620 02:41:09.366207 21152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:09.373976 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21070
W20260620 02:41:09.375754 21132 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42523
W20260620 02:41:09.377420 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21070: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:09.377576 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 21070
2026-06-20T02:41:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2267 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:09 dist-test-slave-rpg6 krb5kdc[21169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:09 dist-test-slave-rpg6 krb5kdc[21169](info): set up 2 sockets
Jun 20 02:41:09 dist-test-slave-rpg6 krb5kdc[21169](info): commencing operation
krb5kdc: starting...
W20260620 02:41:10.636704 388 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:10 dist-test-slave-rpg6 krb5kdc[21169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:11.005970 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:11.461977 21182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:11.462368 21182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:11.462546 21182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:11.470824 21182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:11.471033 21182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:11.471156 21182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:11.471275 21182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:11.471361 21182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:11.483480 21182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:11.485526 21182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:11.487304 21182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:11.503796 21187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:11.503756 21190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:11.503849 21188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:11.505796 21182 server_base.cc:1034] running on GCE node
I20260620 02:41:11.506939 21182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:11.509250 21182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:11.510610 21182 hybrid_clock.cc:648] HybridClock initialized: now 1781923271510559 us; error 50 us; skew 500 ppm
I20260620 02:41:11.511210 21182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:11 dist-test-slave-rpg6 krb5kdc[21169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:11.515609 21182 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:11.518217 21182 webserver.cc:460] Webserver started at http://127.0.0.1:37655/ using document root <none> and password file <none>
I20260620 02:41:11.519083 21182 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:11.519268 21182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:11.519623 21182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:11.523092 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "b617c183e52741d1968d4e6692a66125"
format_stamp: "Formatted at 2026-06-20 02:41:11 on dist-test-slave-rpg6"
server_key: "2f9fb6c18a05785c6d4e7cef8806e49c"
server_key_iv: "d4bdafd8ca0e16a6d95be15a9d974381"
server_key_version: "encryptionkey@0"
I20260620 02:41:11.524076 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "b617c183e52741d1968d4e6692a66125"
format_stamp: "Formatted at 2026-06-20 02:41:11 on dist-test-slave-rpg6"
server_key: "2f9fb6c18a05785c6d4e7cef8806e49c"
server_key_iv: "d4bdafd8ca0e16a6d95be15a9d974381"
server_key_version: "encryptionkey@0"
I20260620 02:41:11.530786 21182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:41:11.537509 21197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:11.539587 21182 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:41:11.539927 21182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "b617c183e52741d1968d4e6692a66125"
format_stamp: "Formatted at 2026-06-20 02:41:11 on dist-test-slave-rpg6"
server_key: "2f9fb6c18a05785c6d4e7cef8806e49c"
server_key_iv: "d4bdafd8ca0e16a6d95be15a9d974381"
server_key_version: "encryptionkey@0"
I20260620 02:41:11.540189 21182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:11.566584 21182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:11.573112 21182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:11.573624 21182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:11.597023 21182 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34677
I20260620 02:41:11.597055 21248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34677 every 8 connection(s)
I20260620 02:41:11.598536 21182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:11.603717 21249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:11.607084 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 21182
I20260620 02:41:11.607304 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:11.607818 388 external_mini_cluster.cc:1428] Setting key 05b59ceba02f5276476456c5a22cceb6
I20260620 02:41:11.618304 21249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125: Bootstrap starting.
I20260620 02:41:11.622295 21249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:11.623636 21249 log.cc:824] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:41:11 dist-test-slave-rpg6 krb5kdc[21169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:41:11.627640 21249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125: No bootstrap required, opened a new log
I20260620 02:41:11.638615 21249 raft_consensus.cc:348] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b617c183e52741d1968d4e6692a66125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34677 } }
I20260620 02:41:11.639712 21249 raft_consensus.cc:374] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:11.639962 21249 raft_consensus.cc:729] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b617c183e52741d1968d4e6692a66125, State: Initialized, Role: FOLLOWER
I20260620 02:41:11.640057 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:11.610605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46855 (local address 127.0.0.1:34677)
0620 02:41:11.611510 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:11.611522 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:11.611574 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:41:11.613622 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:11.613650 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:11.614163 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:11.614661 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:11.617194 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:11.619155 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:11.622127 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:11.623047 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:11.623300 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:11.628986 (+ 5686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:11.628996 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:11.629013 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:11.629058 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:11.634117 (+ 5059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:11.635186 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:11.635195 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:11.635203 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:11.635450 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:11.636064 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:11.636070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:11.636072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:11.636817 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:11.637217 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:11.637803 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 02:41:11.638316 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":387,"threads_started":1}
I20260620 02:41:11.641568 21249 consensus_queue.cc:260] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b617c183e52741d1968d4e6692a66125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34677 } }
I20260620 02:41:11.642202 21249 raft_consensus.cc:388] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:11.642376 21249 raft_consensus.cc:482] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:11.642575 21249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:11.646569 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:11.646699 21249 raft_consensus.cc:504] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b617c183e52741d1968d4e6692a66125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34677 } }
I20260620 02:41:11.647482 21249 leader_election.cc:302] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b617c183e52741d1968d4e6692a66125; no voters:
I20260620 02:41:11.648730 21254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:11.648826 21249 leader_election.cc:288] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:11.649820 21254 raft_consensus.cc:686] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [term 1 LEADER]: Becoming Leader. State: Replica: b617c183e52741d1968d4e6692a66125, State: Running, Role: LEADER
I20260620 02:41:11.650612 21254 consensus_queue.cc:237] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b617c183e52741d1968d4e6692a66125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34677 } }
I20260620 02:41:11.656344 21249 sys_catalog.cc:564] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:11.656951 21255 sys_catalog.cc:455] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b617c183e52741d1968d4e6692a66125. Latest consensus state: current_term: 1 leader_uuid: "b617c183e52741d1968d4e6692a66125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b617c183e52741d1968d4e6692a66125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34677 } } }
I20260620 02:41:11.656873 21256 sys_catalog.cc:455] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b617c183e52741d1968d4e6692a66125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b617c183e52741d1968d4e6692a66125" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34677 } } }
I20260620 02:41:11.657394 21255 sys_catalog.cc:458] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:11.657538 21256 sys_catalog.cc:458] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:11.658537 21263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:11.674154 21263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:11.696584 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:11.658025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56805 (local address 127.0.0.1:34677)
0620 02:41:11.658281 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:11.658288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:11.666313 (+ 8025us) server_negotiation.cc:407] Connection header received
0620 02:41:11.666772 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:11.666780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:11.666923 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:11.667034 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:11.670245 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:11.671500 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:11.673717 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:11.674782 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:11.675149 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:11.681181 (+ 6032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:11.681190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:11.681195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:11.681269 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:11.688006 (+ 6737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:11.689600 (+ 1594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:11.689609 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:11.689615 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:11.689851 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:11.690549 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:11.690557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:11.690563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:11.691237 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:11.694753 (+ 3516us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:11.695050 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 02:41:11.695546 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:41:11.705215 21263 catalog_manager.cc:1347] Generated new cluster ID: 4c2637cb3985409f87f213474727fb8c
I20260620 02:41:11.705384 21263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:11.732710 21263 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:11.735024 21263 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:11.746225 21263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b617c183e52741d1968d4e6692a66125: Generated new TSK 0
I20260620 02:41:11.747437 21263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:11.755559 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21182
I20260620 02:41:11.757736 21244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:11.872468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 21182
2026-06-20T02:41:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:12 dist-test-slave-rpg6 krb5kdc[21281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:12 dist-test-slave-rpg6 krb5kdc[21281](info): set up 2 sockets
Jun 20 02:41:12 dist-test-slave-rpg6 krb5kdc[21281](info): commencing operation
krb5kdc: starting...
W20260620 02:41:13.113281 388 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:13 dist-test-slave-rpg6 krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:13.443341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:13.857525 21294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:13.857877 21294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:13.858052 21294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:13.866075 21294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:13.866236 21294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:13.866447 21294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:13.866606 21294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:13.866758 21294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:13.878293 21294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:13.879868 21294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:13.881487 21294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:13.899464 21300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:13.900039 21299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:13.900144 21302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:13.906683 21294 server_base.cc:1034] running on GCE node
I20260620 02:41:13.907990 21294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:13.909760 21294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:13.911170 21294 hybrid_clock.cc:648] HybridClock initialized: now 1781923273911088 us; error 79 us; skew 500 ppm
I20260620 02:41:13.912171 21294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:13 dist-test-slave-rpg6 krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:13.916491 21294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:13.918522 21294 webserver.cc:460] Webserver started at http://127.0.0.1:41131/ using document root <none> and password file <none>
I20260620 02:41:13.919400 21294 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:13.919576 21294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:13.919997 21294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:13.924887 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "cb09478217c341dfbffc8926c73cf567"
format_stamp: "Formatted at 2026-06-20 02:41:13 on dist-test-slave-rpg6"
server_key: "798f9f9ce31dcd76b790ac518fe624a3"
server_key_iv: "3f06dc36352ab01131c9f8088e347838"
server_key_version: "encryptionkey@0"
I20260620 02:41:13.926585 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "cb09478217c341dfbffc8926c73cf567"
format_stamp: "Formatted at 2026-06-20 02:41:13 on dist-test-slave-rpg6"
server_key: "798f9f9ce31dcd76b790ac518fe624a3"
server_key_iv: "3f06dc36352ab01131c9f8088e347838"
server_key_version: "encryptionkey@0"
I20260620 02:41:13.936503 21294 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260620 02:41:13.941223 21309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:13.943535 21294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:41:13.944098 21294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "cb09478217c341dfbffc8926c73cf567"
format_stamp: "Formatted at 2026-06-20 02:41:13 on dist-test-slave-rpg6"
server_key: "798f9f9ce31dcd76b790ac518fe624a3"
server_key_iv: "3f06dc36352ab01131c9f8088e347838"
server_key_version: "encryptionkey@0"
I20260620 02:41:13.944365 21294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:13.975476 21294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:13.982306 21294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:13.982628 21294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:14.007668 21294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35747
I20260620 02:41:14.007678 21360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35747 every 8 connection(s)
I20260620 02:41:14.009328 21294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:14.014784 21361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:14.018749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 21294
I20260620 02:41:14.018980 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:14.019548 388 external_mini_cluster.cc:1428] Setting key 53a5b5b6c937e75c9dba867ba5cc0e89
I20260620 02:41:14.033179 21361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567: Bootstrap starting.
I20260620 02:41:14.040556 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:14.022449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51077 (local address 127.0.0.1:35747)
0620 02:41:14.023669 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:14.023684 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:14.023744 (+ 60us) server_negotiation.cc:407] Connection header received
0620 02:41:14.025919 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:14.025950 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:14.026541 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:14.027070 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:14.027815 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:14.029415 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:14.031043 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:14.031986 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:14.032340 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:14.034217 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:14.034230 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:14.034254 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:14.035442 (+ 1188us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:14.035665 (+ 223us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:14.035739 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:14.036049 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:14.036706 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 02:41:14.037240 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":490,"threads_started":1}
I20260620 02:41:14.040601 21361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:14.042819 21361 log.cc:824] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:14.048904 21361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567: No bootstrap required, opened a new log
I20260620 02:41:14.049408 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:14.062183 21361 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09478217c341dfbffc8926c73cf567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260620 02:41:14.063427 21361 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:14.063621 21361 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb09478217c341dfbffc8926c73cf567, State: Initialized, Role: FOLLOWER
I20260620 02:41:14.064510 21361 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09478217c341dfbffc8926c73cf567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260620 02:41:14.064882 21361 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:14.065048 21361 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:14.065296 21361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:14.066553 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:14.056097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50981 (local address 127.0.0.1:35747)
0620 02:41:14.056454 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:14.056465 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:14.057619 (+ 1154us) server_negotiation.cc:407] Connection header received
0620 02:41:14.057896 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:14.057902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:14.057983 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:14.058112 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:14.058872 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:14.060379 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:14.063032 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:14.063774 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:14.063976 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:14.065388 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:14.065396 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:14.065401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:14.065434 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:14.065498 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:14.065523 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:14.065653 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:14.066013 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 02:41:14.066143 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260620 02:41:14.068822 21361 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09478217c341dfbffc8926c73cf567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260620 02:41:14.069573 21361 leader_election.cc:302] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb09478217c341dfbffc8926c73cf567; no voters:
I20260620 02:41:14.070526 21361 leader_election.cc:288] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:14.070842 21371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:14.072117 21371 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [term 1 LEADER]: Becoming Leader. State: Replica: cb09478217c341dfbffc8926c73cf567, State: Running, Role: LEADER
I20260620 02:41:14.072712 21371 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09478217c341dfbffc8926c73cf567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260620 02:41:14.073303 21361 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:14.077420 21372 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb09478217c341dfbffc8926c73cf567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09478217c341dfbffc8926c73cf567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } } }
I20260620 02:41:14.077399 21373 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb09478217c341dfbffc8926c73cf567. Latest consensus state: current_term: 1 leader_uuid: "cb09478217c341dfbffc8926c73cf567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09478217c341dfbffc8926c73cf567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } } }
I20260620 02:41:14.078076 21372 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:14.078076 21373 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:14.080320 21379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:14.087882 21379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:14.104980 21379 catalog_manager.cc:1347] Generated new cluster ID: cd19f547a6424f5baa7d65578c115298
I20260620 02:41:14.105158 21379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:14.121083 21379 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:14.122931 21379 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:14.133666 21379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb09478217c341dfbffc8926c73cf567: Generated new TSK 0
I20260620 02:41:14.134828 21379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:14.154908 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21294
W20260620 02:41:14.156700 21356 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51077
W20260620 02:41:14.157609 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21294: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:14.157784 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 21294
2026-06-20T02:41:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:14 dist-test-slave-rpg6 krb5kdc[21392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:14 dist-test-slave-rpg6 krb5kdc[21392](info): set up 2 sockets
Jun 20 02:41:14 dist-test-slave-rpg6 krb5kdc[21392](info): commencing operation
krb5kdc: starting...
W20260620 02:41:15.401324 388 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:15 dist-test-slave-rpg6 krb5kdc[21392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:15.718293 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:16.127702 21405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:16.128201 21405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:16.128309 21405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:16.136183 21405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:16.136354 21405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:16.136406 21405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:16.136471 21405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:16.136529 21405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:16.149529 21405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:16.151172 21405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:16.152640 21405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:16.166076 21410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:16.165973 21413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:16.166050 21411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:16.167228 21405 server_base.cc:1034] running on GCE node
I20260620 02:41:16.168387 21405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:16.170254 21405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:16.173746 21405 hybrid_clock.cc:648] HybridClock initialized: now 1781923276173636 us; error 93 us; skew 500 ppm
I20260620 02:41:16.174574 21405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:16 dist-test-slave-rpg6 krb5kdc[21392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:16.178689 21405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:16.180954 21405 webserver.cc:460] Webserver started at http://127.0.0.1:36083/ using document root <none> and password file <none>
I20260620 02:41:16.181779 21405 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:16.181967 21405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:16.182350 21405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:16.186893 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "ff14907af6174c79a80f491aa33a9c28"
format_stamp: "Formatted at 2026-06-20 02:41:16 on dist-test-slave-rpg6"
server_key: "ef2c67b5d1541342e7fdd55768d4e844"
server_key_iv: "7fc8ea06cb2f750a2bca2627cde7f8ac"
server_key_version: "encryptionkey@0"
I20260620 02:41:16.188041 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff14907af6174c79a80f491aa33a9c28"
format_stamp: "Formatted at 2026-06-20 02:41:16 on dist-test-slave-rpg6"
server_key: "ef2c67b5d1541342e7fdd55768d4e844"
server_key_iv: "7fc8ea06cb2f750a2bca2627cde7f8ac"
server_key_version: "encryptionkey@0"
I20260620 02:41:16.196516 21405 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 02:41:16.201869 21420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:16.204361 21405 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:41:16.205070 21405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "ff14907af6174c79a80f491aa33a9c28"
format_stamp: "Formatted at 2026-06-20 02:41:16 on dist-test-slave-rpg6"
server_key: "ef2c67b5d1541342e7fdd55768d4e844"
server_key_iv: "7fc8ea06cb2f750a2bca2627cde7f8ac"
server_key_version: "encryptionkey@0"
I20260620 02:41:16.205406 21405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:16.212813 21405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:16.218971 21405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:16.219343 21405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:16.243693 21405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36775
I20260620 02:41:16.244014 21471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36775 every 8 connection(s)
I20260620 02:41:16.245412 21405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:16.250037 21472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:16.252737 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 21405
I20260620 02:41:16.253007 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:16.253463 388 external_mini_cluster.cc:1428] Setting key c5064d9ffb7e3968cdd7ff7d42fec26e
I20260620 02:41:16.263128 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:16.255549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42699 (local address 127.0.0.1:36775)
0620 02:41:16.256534 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:16.256548 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:16.256598 (+ 50us) server_negotiation.cc:407] Connection header received
0620 02:41:16.259066 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:16.259097 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:16.259513 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:16.260018 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:16.261247 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:16.261254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:16.261275 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:16.261290 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:16.261359 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:16.261381 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:16.261663 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:16.261832 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 02:41:16.262062 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":452,"threads_started":1}
I20260620 02:41:16.264511 21472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28: Bootstrap starting.
I20260620 02:41:16.268028 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:16.269326 21472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:16.271189 21472 log.cc:824] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:16.276000 21472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28: No bootstrap required, opened a new log
I20260620 02:41:16.277475 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:16.273841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49137 (local address 127.0.0.1:36775)
0620 02:41:16.274075 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:16.274083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:16.274450 (+ 367us) server_negotiation.cc:407] Connection header received
0620 02:41:16.274649 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:16.274654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:16.274742 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:16.274911 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:16.276693 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:16.276698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:16.276704 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:16.276754 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:16.276810 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:16.276838 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:16.276990 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:16.277100 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 02:41:16.277166 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 02:41:16.286996 21472 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff14907af6174c79a80f491aa33a9c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 02:41:16.288403 21472 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:16.288630 21472 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff14907af6174c79a80f491aa33a9c28, State: Initialized, Role: FOLLOWER
I20260620 02:41:16.289888 21472 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff14907af6174c79a80f491aa33a9c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 02:41:16.290341 21472 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:16.290624 21472 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:16.291154 21472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:16.295133 21472 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff14907af6174c79a80f491aa33a9c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 02:41:16.295987 21472 leader_election.cc:302] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff14907af6174c79a80f491aa33a9c28; no voters:
I20260620 02:41:16.296845 21472 leader_election.cc:288] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:16.297058 21482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:16.300786 21482 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [term 1 LEADER]: Becoming Leader. State: Replica: ff14907af6174c79a80f491aa33a9c28, State: Running, Role: LEADER
I20260620 02:41:16.301640 21482 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff14907af6174c79a80f491aa33a9c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260620 02:41:16.303035 21472 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:16.305749 21484 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff14907af6174c79a80f491aa33a9c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff14907af6174c79a80f491aa33a9c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260620 02:41:16.306483 21484 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:16.307641 21483 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff14907af6174c79a80f491aa33a9c28. Latest consensus state: current_term: 1 leader_uuid: "ff14907af6174c79a80f491aa33a9c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff14907af6174c79a80f491aa33a9c28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260620 02:41:16.308139 21483 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:16.313212 21489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:16.321118 21489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:16.333423 21489 catalog_manager.cc:1347] Generated new cluster ID: fc8af9c283fb42339a4359c0e5884b12
I20260620 02:41:16.333575 21489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:16.354260 21489 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:16.354815 21489 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:16.367702 21489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff14907af6174c79a80f491aa33a9c28: Generated new TSK 0
I20260620 02:41:16.368757 21489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:16.389075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21405
W20260620 02:41:16.390583 21467 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42699
W20260620 02:41:16.391445 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21405: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:16.391626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 21405
2026-06-20T02:41:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2231 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:16 dist-test-slave-rpg6 krb5kdc[21503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:16 dist-test-slave-rpg6 krb5kdc[21503](info): set up 2 sockets
Jun 20 02:41:16 dist-test-slave-rpg6 krb5kdc[21503](info): commencing operation
krb5kdc: starting...
W20260620 02:41:17.577167 388 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.005s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:17 dist-test-slave-rpg6 krb5kdc[21503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:17.912664 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:18.322672 21516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:18.323024 21516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:18.323194 21516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:18.331499 21516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:18.331678 21516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:18.331741 21516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:18.331816 21516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:18.331955 21516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:18.346522 21516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:18.348124 21516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:18.349643 21516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:18.364280 21524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:18.364347 21522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:18.364316 21521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:18.368065 21516 server_base.cc:1034] running on GCE node
I20260620 02:41:18.369482 21516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:18.371268 21516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:18.372548 21516 hybrid_clock.cc:648] HybridClock initialized: now 1781923278372491 us; error 60 us; skew 500 ppm
I20260620 02:41:18.373018 21516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:18 dist-test-slave-rpg6 krb5kdc[21503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:18.381369 21516 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:18.383368 21516 webserver.cc:460] Webserver started at http://127.0.0.1:45421/ using document root <none> and password file <none>
I20260620 02:41:18.384251 21516 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:18.384418 21516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:18.384756 21516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:18.388293 21516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "0dafe7178de5439daf86e82dcdfab561"
format_stamp: "Formatted at 2026-06-20 02:41:18 on dist-test-slave-rpg6"
server_key: "f370fd0eb421d11ae922ea9bf746aec0"
server_key_iv: "d285f24f6a74ca065ccffce30265f7c5"
server_key_version: "encryptionkey@0"
I20260620 02:41:18.389206 21516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "0dafe7178de5439daf86e82dcdfab561"
format_stamp: "Formatted at 2026-06-20 02:41:18 on dist-test-slave-rpg6"
server_key: "f370fd0eb421d11ae922ea9bf746aec0"
server_key_iv: "d285f24f6a74ca065ccffce30265f7c5"
server_key_version: "encryptionkey@0"
I20260620 02:41:18.396374 21516 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 02:41:18.401702 21531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:18.403729 21516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:41:18.404023 21516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "0dafe7178de5439daf86e82dcdfab561"
format_stamp: "Formatted at 2026-06-20 02:41:18 on dist-test-slave-rpg6"
server_key: "f370fd0eb421d11ae922ea9bf746aec0"
server_key_iv: "d285f24f6a74ca065ccffce30265f7c5"
server_key_version: "encryptionkey@0"
I20260620 02:41:18.404284 21516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:18.425836 21516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:18.432467 21516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:18.432860 21516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:18.457595 21582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34855 every 8 connection(s)
I20260620 02:41:18.457589 21516 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34855
I20260620 02:41:18.460489 21516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:18.465267 21583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:18.468094 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 21516
I20260620 02:41:18.468344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:18.468945 388 external_mini_cluster.cc:1428] Setting key d95ad7249e0bfb30c308c0b1dd6c84ea
I20260620 02:41:18.479749 21583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561: Bootstrap starting.
I20260620 02:41:18.484607 21583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:18.486577 21583 log.cc:824] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:41:18 dist-test-slave-rpg6 krb5kdc[21503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:41:18.490797 21583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561: No bootstrap required, opened a new log
I20260620 02:41:18.502040 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:18.471303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:49525 (local address 127.0.0.1:34855)
0620 02:41:18.472261 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:18.472274 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:18.472321 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:41:18.474334 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:18.474360 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:18.474860 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:18.475259 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:18.477703 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:18.479423 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:18.481762 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:18.482540 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:18.482716 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:18.489872 (+ 7156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:18.489883 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:18.489900 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:18.490270 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:18.495525 (+ 5255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:18.497410 (+ 1885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:18.497422 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:18.497431 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:18.497663 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:18.498262 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:18.498267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:18.498270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:18.499084 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:18.499364 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:18.500301 (+ 937us) server_negotiation.cc:299] Negotiation successful
0620 02:41:18.500640 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":456,"threads_started":1}
I20260620 02:41:18.503324 21583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dafe7178de5439daf86e82dcdfab561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34855 } }
I20260620 02:41:18.504273 21583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:18.504446 21583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dafe7178de5439daf86e82dcdfab561, State: Initialized, Role: FOLLOWER
I20260620 02:41:18.505190 21583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dafe7178de5439daf86e82dcdfab561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34855 } }
I20260620 02:41:18.505586 21583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:18.505978 21583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:18.506255 21583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:18.508394 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:18.509644 21583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dafe7178de5439daf86e82dcdfab561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34855 } }
I20260620 02:41:18.510345 21583 leader_election.cc:302] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dafe7178de5439daf86e82dcdfab561; no voters:
I20260620 02:41:18.511224 21583 leader_election.cc:288] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:18.511900 21588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:18.513744 21588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [term 1 LEADER]: Becoming Leader. State: Replica: 0dafe7178de5439daf86e82dcdfab561, State: Running, Role: LEADER
I20260620 02:41:18.514565 21588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dafe7178de5439daf86e82dcdfab561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34855 } }
I20260620 02:41:18.515050 21583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:18.519054 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dafe7178de5439daf86e82dcdfab561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dafe7178de5439daf86e82dcdfab561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34855 } } }
I20260620 02:41:18.519524 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:18.520256 21592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dafe7178de5439daf86e82dcdfab561. Latest consensus state: current_term: 1 leader_uuid: "0dafe7178de5439daf86e82dcdfab561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dafe7178de5439daf86e82dcdfab561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34855 } } }
I20260620 02:41:18.520584 21592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:18.535655 21599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:18.552434 21599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:18.552838 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:18.516085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:59063 (local address 127.0.0.1:34855)
0620 02:41:18.516308 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:18.516314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:18.516775 (+ 461us) server_negotiation.cc:407] Connection header received
0620 02:41:18.517244 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:18.517253 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:18.517387 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:18.517550 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:18.519942 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:18.521738 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:18.523490 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:18.524507 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:18.524734 (+ 227us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:18.531412 (+ 6678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:18.531426 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:18.531434 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:18.531513 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:18.540366 (+ 8853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:18.542750 (+ 2384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:18.542761 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:18.542767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:18.543025 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:18.544170 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:18.544180 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:18.544186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:18.544769 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:18.545176 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:18.546977 (+ 1801us) server_negotiation.cc:299] Negotiation successful
0620 02:41:18.547286 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 02:41:18.570912 21599 catalog_manager.cc:1347] Generated new cluster ID: 185839c20e4e44a2a7cba92c9071126d
I20260620 02:41:18.571063 21599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:18.610002 21599 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:18.612869 21599 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:18.626561 21599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dafe7178de5439daf86e82dcdfab561: Generated new TSK 0
I20260620 02:41:18.627732 21599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:18.637437 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21516
I20260620 02:41:18.638875 21578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:18.745568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 21516
W20260620 02:41:18.767064 21501 connection.cc:535] client connection to 127.0.0.1:34855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34855): Connection reset by peer (error 104)
2026-06-20T02:41:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:18 dist-test-slave-rpg6 krb5kdc[21616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:18 dist-test-slave-rpg6 krb5kdc[21616](info): set up 2 sockets
Jun 20 02:41:18 dist-test-slave-rpg6 krb5kdc[21616](info): commencing operation
krb5kdc: starting...
W20260620 02:41:19.980643 388 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:20 dist-test-slave-rpg6 krb5kdc[21616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:20.334749 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:20.752609 21629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:20.752957 21629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:20.753053 21629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:20.763748 21629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:20.764009 21629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:20.764072 21629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:20.764138 21629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:20.764194 21629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:20.779559 21629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:20.781008 21629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:20.782429 21629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:20.799432 21637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:20.799403 21634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:20.799592 21635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:20.801214 21629 server_base.cc:1034] running on GCE node
I20260620 02:41:20.802234 21629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:20.804041 21629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:20.805502 21629 hybrid_clock.cc:648] HybridClock initialized: now 1781923280805413 us; error 87 us; skew 500 ppm
I20260620 02:41:20.805941 21629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:20 dist-test-slave-rpg6 krb5kdc[21616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:20.810282 21629 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:20.813556 21629 webserver.cc:460] Webserver started at http://127.0.0.1:38335/ using document root <none> and password file <none>
I20260620 02:41:20.814692 21629 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:20.814878 21629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:20.815243 21629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:20.819379 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "f2aab84980cf46ca911abe9ba205c223"
format_stamp: "Formatted at 2026-06-20 02:41:20 on dist-test-slave-rpg6"
server_key: "aa0394dc2958a4d8f008e26b4ba5182b"
server_key_iv: "09ab334448c58db9f185cd98db61e82b"
server_key_version: "encryptionkey@0"
I20260620 02:41:20.820613 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "f2aab84980cf46ca911abe9ba205c223"
format_stamp: "Formatted at 2026-06-20 02:41:20 on dist-test-slave-rpg6"
server_key: "aa0394dc2958a4d8f008e26b4ba5182b"
server_key_iv: "09ab334448c58db9f185cd98db61e82b"
server_key_version: "encryptionkey@0"
I20260620 02:41:20.830149 21629 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260620 02:41:20.836392 21644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:20.838246 21629 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 02:41:20.838636 21629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "f2aab84980cf46ca911abe9ba205c223"
format_stamp: "Formatted at 2026-06-20 02:41:20 on dist-test-slave-rpg6"
server_key: "aa0394dc2958a4d8f008e26b4ba5182b"
server_key_iv: "09ab334448c58db9f185cd98db61e82b"
server_key_version: "encryptionkey@0"
I20260620 02:41:20.838933 21629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:20.868672 21629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:20.875473 21629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:20.875874 21629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:20.902050 21629 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34425
I20260620 02:41:20.902143 21695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34425 every 8 connection(s)
I20260620 02:41:20.904121 21629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:20.909117 21696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:20.913044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 21629
I20260620 02:41:20.913290 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:20.913727 388 external_mini_cluster.cc:1428] Setting key 8029bef603728ef2da22c841618f3201
I20260620 02:41:20.926674 21696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223: Bootstrap starting.
I20260620 02:41:20.930979 21696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:20.932519 21696 log.cc:824] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:20.934191 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:20.916272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:60015 (local address 127.0.0.1:34425)
0620 02:41:20.917627 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:20.917651 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:20.917729 (+ 78us) server_negotiation.cc:407] Connection header received
0620 02:41:20.920931 (+ 3202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:20.920990 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:20.921983 (+ 993us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:20.923033 (+ 1050us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:20.923753 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:20.925885 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:20.927591 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:20.928822 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:20.929064 (+ 242us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:20.930868 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:20.930881 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:20.930905 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:20.931728 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:20.931880 (+ 152us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:20.931917 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:20.932100 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:20.932471 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 02:41:20.932805 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":637,"threads_started":1}
I20260620 02:41:20.936277 21696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223: No bootstrap required, opened a new log
I20260620 02:41:20.940083 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:20.946117 21696 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aab84980cf46ca911abe9ba205c223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } }
I20260620 02:41:20.947351 21696 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:20.947785 21696 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aab84980cf46ca911abe9ba205c223, State: Initialized, Role: FOLLOWER
I20260620 02:41:20.948755 21696 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aab84980cf46ca911abe9ba205c223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } }
I20260620 02:41:20.949105 21696 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:20.949277 21696 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:20.949524 21696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:20.952160 21696 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aab84980cf46ca911abe9ba205c223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } }
I20260620 02:41:20.953446 21696 leader_election.cc:302] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2aab84980cf46ca911abe9ba205c223; no voters:
I20260620 02:41:20.954273 21696 leader_election.cc:288] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:20.954403 21706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:20.955350 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:20.946543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:54901 (local address 127.0.0.1:34425)
0620 02:41:20.946918 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:20.946926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:20.947633 (+ 707us) server_negotiation.cc:407] Connection header received
0620 02:41:20.947946 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:20.947957 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:20.948067 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:20.948224 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:20.948717 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:20.950056 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:20.951534 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:20.952130 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:20.952773 (+ 643us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:20.954349 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:20.954373 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:20.954380 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:20.954410 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:20.954493 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:20.954524 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:20.954804 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:20.954987 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 02:41:20.955047 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:41:20.962297 21706 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [term 1 LEADER]: Becoming Leader. State: Replica: f2aab84980cf46ca911abe9ba205c223, State: Running, Role: LEADER
I20260620 02:41:20.962879 21696 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:20.963438 21706 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aab84980cf46ca911abe9ba205c223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } }
I20260620 02:41:20.967137 21707 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2aab84980cf46ca911abe9ba205c223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aab84980cf46ca911abe9ba205c223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } } }
I20260620 02:41:20.967589 21707 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:20.968206 21708 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2aab84980cf46ca911abe9ba205c223. Latest consensus state: current_term: 1 leader_uuid: "f2aab84980cf46ca911abe9ba205c223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aab84980cf46ca911abe9ba205c223" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } } }
I20260620 02:41:20.968554 21708 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:20.971338 21715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:20.978991 21715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:20.993060 21715 catalog_manager.cc:1347] Generated new cluster ID: 8d403ebb9fa84b80965b977114187b92
I20260620 02:41:20.993220 21715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:21.006994 21715 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:21.009305 21715 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:21.027455 21715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2aab84980cf46ca911abe9ba205c223: Generated new TSK 0
I20260620 02:41:21.028581 21715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:21.045270 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21629
W20260620 02:41:21.047191 21691 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.93:60015
W20260620 02:41:21.048648 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21629: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:21.049134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 21629
W20260620 02:41:21.076014 21614 connection.cc:535] client connection to 127.0.0.1:34425 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34425): Connection reset by peer (error 104)
2026-06-20T02:41:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:21 dist-test-slave-rpg6 krb5kdc[21728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:21 dist-test-slave-rpg6 krb5kdc[21728](info): set up 2 sockets
Jun 20 02:41:21 dist-test-slave-rpg6 krb5kdc[21728](info): commencing operation
krb5kdc: starting...
W20260620 02:41:22.248562 388 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:22 dist-test-slave-rpg6 krb5kdc[21728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:22.573997 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:23.002120 21741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:23.002477 21741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:23.002669 21741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:23.010732 21741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:23.010887 21741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:23.010936 21741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:23.011066 21741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:23.011127 21741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:23.023383 21741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:23.025784 21741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:23.027336 21741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:23.045356 21746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:23.045356 21749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:23.046805 21747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:23.047607 21741 server_base.cc:1034] running on GCE node
I20260620 02:41:23.049464 21741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:23.051397 21741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:23.053736 21741 hybrid_clock.cc:648] HybridClock initialized: now 1781923283053522 us; error 203 us; skew 500 ppm
I20260620 02:41:23.054232 21741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:23 dist-test-slave-rpg6 krb5kdc[21728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:23.060078 21741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:23.063525 21741 webserver.cc:460] Webserver started at http://127.0.0.1:33267/ using document root <none> and password file <none>
I20260620 02:41:23.064886 21741 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:23.065258 21741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:23.066273 21741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:23.071563 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "22e330b1d3b74578be2938812f08287d"
format_stamp: "Formatted at 2026-06-20 02:41:23 on dist-test-slave-rpg6"
server_key: "ab2db854936a50123276fceae8233d49"
server_key_iv: "136a2d86287343deb7078f43a87b86fc"
server_key_version: "encryptionkey@0"
I20260620 02:41:23.072705 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "22e330b1d3b74578be2938812f08287d"
format_stamp: "Formatted at 2026-06-20 02:41:23 on dist-test-slave-rpg6"
server_key: "ab2db854936a50123276fceae8233d49"
server_key_iv: "136a2d86287343deb7078f43a87b86fc"
server_key_version: "encryptionkey@0"
I20260620 02:41:23.081506 21741 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260620 02:41:23.088021 21756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:23.091281 21741 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260620 02:41:23.091574 21741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "22e330b1d3b74578be2938812f08287d"
format_stamp: "Formatted at 2026-06-20 02:41:23 on dist-test-slave-rpg6"
server_key: "ab2db854936a50123276fceae8233d49"
server_key_iv: "136a2d86287343deb7078f43a87b86fc"
server_key_version: "encryptionkey@0"
I20260620 02:41:23.092407 21741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:23.103048 21741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:23.110334 21741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:23.110841 21741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:23.139997 21741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36105
I20260620 02:41:23.140013 21807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36105 every 8 connection(s)
I20260620 02:41:23.141691 21741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:23.146648 21808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:23.149014 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 21741
I20260620 02:41:23.149227 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:23.149690 388 external_mini_cluster.cc:1428] Setting key 8107927eb9407a38185cd6c0c2091763
I20260620 02:41:23.159596 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:23.151902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:41349 (local address 127.0.0.1:36105)
0620 02:41:23.152962 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:23.152976 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:23.153054 (+ 78us) server_negotiation.cc:407] Connection header received
0620 02:41:23.154938 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:23.154972 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:23.155441 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:23.155982 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:23.157125 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:23.157131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:23.157145 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:23.157161 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:23.157372 (+ 211us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:23.157504 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:23.158142 (+ 638us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:23.158341 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 02:41:23.158526 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":477,"threads_started":1}
I20260620 02:41:23.163920 21808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d: Bootstrap starting.
I20260620 02:41:23.164885 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:23.167711 21808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:23.169142 21808 log.cc:824] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:23.173166 21808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d: No bootstrap required, opened a new log
I20260620 02:41:23.174031 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:23.170609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:57447 (local address 127.0.0.1:36105)
0620 02:41:23.170807 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:23.170813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:23.171405 (+ 592us) server_negotiation.cc:407] Connection header received
0620 02:41:23.171534 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:23.171538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:23.171597 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:23.171773 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:23.173015 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:23.173022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:23.173026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:23.173054 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:23.173126 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:23.173154 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:23.173399 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:23.173509 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 02:41:23.173592 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:41:23.182615 21808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e330b1d3b74578be2938812f08287d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36105 } }
I20260620 02:41:23.183097 21808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:23.183230 21808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22e330b1d3b74578be2938812f08287d, State: Initialized, Role: FOLLOWER
I20260620 02:41:23.183993 21808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e330b1d3b74578be2938812f08287d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36105 } }
I20260620 02:41:23.184238 21808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:23.184401 21808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:23.184576 21808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:23.188252 21808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e330b1d3b74578be2938812f08287d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36105 } }
I20260620 02:41:23.188922 21808 leader_election.cc:302] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22e330b1d3b74578be2938812f08287d; no voters:
I20260620 02:41:23.189568 21808 leader_election.cc:288] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:23.189800 21818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:23.190534 21818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [term 1 LEADER]: Becoming Leader. State: Replica: 22e330b1d3b74578be2938812f08287d, State: Running, Role: LEADER
I20260620 02:41:23.191138 21818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e330b1d3b74578be2938812f08287d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36105 } }
I20260620 02:41:23.191761 21808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:23.194551 21820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22e330b1d3b74578be2938812f08287d. Latest consensus state: current_term: 1 leader_uuid: "22e330b1d3b74578be2938812f08287d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e330b1d3b74578be2938812f08287d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36105 } } }
I20260620 02:41:23.194604 21819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22e330b1d3b74578be2938812f08287d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e330b1d3b74578be2938812f08287d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36105 } } }
I20260620 02:41:23.195235 21820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:23.195290 21819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:23.196509 21823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:23.205271 21823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:23.221875 21823 catalog_manager.cc:1347] Generated new cluster ID: 73d115503cec4dbcbea6f294734a82da
I20260620 02:41:23.222030 21823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:23.242337 21823 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:23.243234 21823 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:23.252430 21823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22e330b1d3b74578be2938812f08287d: Generated new TSK 0
I20260620 02:41:23.253911 21823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:23.272816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21741
W20260620 02:41:23.274374 21803 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.93:41349
W20260620 02:41:23.275342 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21741: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:23.275547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 21741
2026-06-20T02:41:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:23 dist-test-slave-rpg6 krb5kdc[21839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:23 dist-test-slave-rpg6 krb5kdc[21839](info): set up 2 sockets
Jun 20 02:41:23 dist-test-slave-rpg6 krb5kdc[21839](info): commencing operation
krb5kdc: starting...
W20260620 02:41:24.505165 388 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.003s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:24 dist-test-slave-rpg6 krb5kdc[21839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:24.816887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:25.224095 21852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:25.224478 21852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:25.224637 21852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:25.234030 21852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:25.234366 21852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:25.234491 21852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:25.234627 21852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:25.234719 21852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:25.246855 21852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36779
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:25.249081 21852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:25.251361 21852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:25.267166 21860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:25.267356 21858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:25.267136 21857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:25.270012 21852 server_base.cc:1034] running on GCE node
I20260620 02:41:25.270999 21852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:25.273005 21852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:25.274403 21852 hybrid_clock.cc:648] HybridClock initialized: now 1781923285274325 us; error 65 us; skew 500 ppm
I20260620 02:41:25.274945 21852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:25 dist-test-slave-rpg6 krb5kdc[21839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:25.282972 21852 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:25.285059 21852 webserver.cc:460] Webserver started at http://127.0.0.1:46347/ using document root <none> and password file <none>
I20260620 02:41:25.285938 21852 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:25.286064 21852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:25.286475 21852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:25.290134 21852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "6e6db984ed274e94ac556500095eb444"
format_stamp: "Formatted at 2026-06-20 02:41:25 on dist-test-slave-rpg6"
server_key: "3ef396730f68d3dbe16292c49ffe6d2f"
server_key_iv: "64457e798157d36ad283dc2d88fef011"
server_key_version: "encryptionkey@0"
I20260620 02:41:25.291069 21852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "6e6db984ed274e94ac556500095eb444"
format_stamp: "Formatted at 2026-06-20 02:41:25 on dist-test-slave-rpg6"
server_key: "3ef396730f68d3dbe16292c49ffe6d2f"
server_key_iv: "64457e798157d36ad283dc2d88fef011"
server_key_version: "encryptionkey@0"
I20260620 02:41:25.298460 21852 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260620 02:41:25.304306 21867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:25.306623 21852 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:41:25.306885 21852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "6e6db984ed274e94ac556500095eb444"
format_stamp: "Formatted at 2026-06-20 02:41:25 on dist-test-slave-rpg6"
server_key: "3ef396730f68d3dbe16292c49ffe6d2f"
server_key_iv: "64457e798157d36ad283dc2d88fef011"
server_key_version: "encryptionkey@0"
I20260620 02:41:25.307147 21852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:25.349421 21852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:25.356382 21852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:25.356765 21852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:25.380873 21852 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36779
I20260620 02:41:25.380926 21918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36779 every 8 connection(s)
I20260620 02:41:25.382472 21852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:25.385226 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 21852
I20260620 02:41:25.385425 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:25.385967 388 external_mini_cluster.cc:1428] Setting key 14d9bc592542f9f1cb48b8eeb5d44705
I20260620 02:41:25.387984 21919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:25.406880 21919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444: Bootstrap starting.
Jun 20 02:41:25 dist-test-slave-rpg6 krb5kdc[21839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:41:25.412653 21919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:25.414634 21919 log.cc:824] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:25.418818 21919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444: No bootstrap required, opened a new log
I20260620 02:41:25.421090 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:25.388523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:38179 (local address 127.0.0.1:36779)
0620 02:41:25.389912 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:25.389930 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:25.390007 (+ 77us) server_negotiation.cc:407] Connection header received
0620 02:41:25.392946 (+ 2939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:25.392994 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:25.393937 (+ 943us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:25.394434 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:25.397310 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:25.399161 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:25.401810 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:25.402719 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:25.402923 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:25.409097 (+ 6174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:25.409108 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:25.409127 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:25.409191 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:25.413776 (+ 4585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:25.415364 (+ 1588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:25.415382 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:25.415392 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:25.416235 (+ 843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:25.417518 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:25.417524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:25.417527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:25.418212 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:25.418638 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:25.419448 (+ 810us) server_negotiation.cc:299] Negotiation successful
0620 02:41:25.419892 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":601,"threads_started":1}
I20260620 02:41:25.427295 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:25.429221 21919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6db984ed274e94ac556500095eb444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36779 } }
I20260620 02:41:25.429733 21919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:25.429875 21919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6db984ed274e94ac556500095eb444, State: Initialized, Role: FOLLOWER
I20260620 02:41:25.430612 21919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6db984ed274e94ac556500095eb444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36779 } }
I20260620 02:41:25.430893 21919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:25.431085 21919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:25.431309 21919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:25.434450 21919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6db984ed274e94ac556500095eb444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36779 } }
I20260620 02:41:25.435110 21919 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e6db984ed274e94ac556500095eb444; no voters:
I20260620 02:41:25.435878 21919 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:25.436003 21929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:25.438983 21919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:25.440218 21929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [term 1 LEADER]: Becoming Leader. State: Replica: 6e6db984ed274e94ac556500095eb444, State: Running, Role: LEADER
I20260620 02:41:25.441123 21929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6db984ed274e94ac556500095eb444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36779 } }
I20260620 02:41:25.445084 21931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6db984ed274e94ac556500095eb444. Latest consensus state: current_term: 1 leader_uuid: "6e6db984ed274e94ac556500095eb444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6db984ed274e94ac556500095eb444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36779 } } }
I20260620 02:41:25.445536 21931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:25.450387 21930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e6db984ed274e94ac556500095eb444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6db984ed274e94ac556500095eb444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36779 } } }
I20260620 02:41:25.450835 21930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:25.451849 21936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:25.454505 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:25.433476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:46055 (local address 127.0.0.1:36779)
0620 02:41:25.433695 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:25.433702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:25.434415 (+ 713us) server_negotiation.cc:407] Connection header received
0620 02:41:25.434624 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:25.434630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:25.434762 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:25.434908 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:25.436887 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:25.437848 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:25.439417 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:25.440233 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:25.440401 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:25.443856 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:25.443868 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:25.443879 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:25.443951 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:25.449157 (+ 5206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:25.450689 (+ 1532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:25.450697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:25.450704 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:25.451211 (+ 507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:25.452196 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:25.452207 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:25.452213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:25.452689 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:25.453152 (+ 463us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:25.453546 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 02:41:25.453866 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":62}
I20260620 02:41:25.458748 21936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:25.470819 21936 catalog_manager.cc:1347] Generated new cluster ID: 8d8d1c7bc4084fa68a0908385fc38184
I20260620 02:41:25.470970 21936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:25.485108 21936 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:25.486651 21936 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:25.498056 21936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e6db984ed274e94ac556500095eb444: Generated new TSK 0
I20260620 02:41:25.499256 21936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:25.513806 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21852
I20260620 02:41:25.515858 21914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:25.615159 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 21852
W20260620 02:41:25.636129 21837 connection.cc:535] client connection to 127.0.0.1:36779 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36779): Connection reset by peer (error 104)
2026-06-20T02:41:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:25 dist-test-slave-rpg6 krb5kdc[21951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:25 dist-test-slave-rpg6 krb5kdc[21951](info): set up 2 sockets
Jun 20 02:41:25 dist-test-slave-rpg6 krb5kdc[21951](info): commencing operation
krb5kdc: starting...
W20260620 02:41:26.817234 388 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:27 dist-test-slave-rpg6 krb5kdc[21951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:27.146242 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:27.548780 21964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:27.549180 21964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:27.549347 21964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:27.557505 21964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:27.557688 21964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:27.557801 21964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:27.557901 21964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:27.557974 21964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:27.568738 21964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:27.570225 21964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:27.571683 21964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:27.585552 21969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:27.585572 21972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:27.585671 21970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:27.586776 21964 server_base.cc:1034] running on GCE node
I20260620 02:41:27.587766 21964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:27.589650 21964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:27.591109 21964 hybrid_clock.cc:648] HybridClock initialized: now 1781923287591025 us; error 82 us; skew 500 ppm
I20260620 02:41:27.591572 21964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:27 dist-test-slave-rpg6 krb5kdc[21951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:27.596522 21964 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:27.599011 21964 webserver.cc:460] Webserver started at http://127.0.0.1:35107/ using document root <none> and password file <none>
I20260620 02:41:27.599954 21964 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:27.600131 21964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:27.600481 21964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:27.604197 21964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "8ec36eaaaba14fd7a445e32f8049e2d0"
format_stamp: "Formatted at 2026-06-20 02:41:27 on dist-test-slave-rpg6"
server_key: "a092c210c556ea3feccde576b5cdb451"
server_key_iv: "257a77c987546bf0c17015907b280504"
server_key_version: "encryptionkey@0"
I20260620 02:41:27.605214 21964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ec36eaaaba14fd7a445e32f8049e2d0"
format_stamp: "Formatted at 2026-06-20 02:41:27 on dist-test-slave-rpg6"
server_key: "a092c210c556ea3feccde576b5cdb451"
server_key_iv: "257a77c987546bf0c17015907b280504"
server_key_version: "encryptionkey@0"
I20260620 02:41:27.612249 21964 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260620 02:41:27.617601 21979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:27.620152 21964 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:41:27.620421 21964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "8ec36eaaaba14fd7a445e32f8049e2d0"
format_stamp: "Formatted at 2026-06-20 02:41:27 on dist-test-slave-rpg6"
server_key: "a092c210c556ea3feccde576b5cdb451"
server_key_iv: "257a77c987546bf0c17015907b280504"
server_key_version: "encryptionkey@0"
I20260620 02:41:27.620671 21964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:27.652344 21964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:27.659493 21964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:27.659902 21964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:27.686772 21964 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42885
I20260620 02:41:27.686793 22030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42885 every 8 connection(s)
I20260620 02:41:27.689216 21964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:27.692667 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 21964
I20260620 02:41:27.693192 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:27.693938 388 external_mini_cluster.cc:1428] Setting key 8ab8e83aef7cc015c6e7cf5c9fe79e7b
I20260620 02:41:27.694607 22031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:27.714890 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:27.696021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:47401 (local address 127.0.0.1:42885)
0620 02:41:27.696989 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:27.697002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:27.697065 (+ 63us) server_negotiation.cc:407] Connection header received
0620 02:41:27.700790 (+ 3725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:27.700853 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:27.701820 (+ 967us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:27.702384 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:27.703039 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:27.705290 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:27.707581 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:27.709218 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:27.709559 (+ 341us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:27.711685 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:27.711695 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:27.711720 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:27.712480 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:27.712583 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:27.712615 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:27.712826 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:27.713161 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 02:41:27.713369 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":375,"threads_started":1}
I20260620 02:41:27.714893 22031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0: Bootstrap starting.
I20260620 02:41:27.718916 22031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:27.720772 22031 log.cc:824] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:27.720976 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:27.725015 22031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0: No bootstrap required, opened a new log
I20260620 02:41:27.736251 22031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36eaaaba14fd7a445e32f8049e2d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42885 } }
I20260620 02:41:27.736940 22031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:27.737105 22031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec36eaaaba14fd7a445e32f8049e2d0, State: Initialized, Role: FOLLOWER
I20260620 02:41:27.737429 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:27.726753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:43793 (local address 127.0.0.1:42885)
0620 02:41:27.726989 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:27.726997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:27.727382 (+ 385us) server_negotiation.cc:407] Connection header received
0620 02:41:27.727554 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:27.727560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:27.727656 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:27.727809 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:27.729076 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:27.730396 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:27.732868 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:27.733645 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:27.733839 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:27.736140 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:27.736148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:27.736153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:27.736182 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:27.736284 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:27.736313 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:27.736472 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:27.737019 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 02:41:27.737089 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:41:27.738022 22031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36eaaaba14fd7a445e32f8049e2d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42885 } }
I20260620 02:41:27.738410 22031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:27.738605 22031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:27.738804 22031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:27.741714 22031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36eaaaba14fd7a445e32f8049e2d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42885 } }
I20260620 02:41:27.742506 22031 leader_election.cc:302] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ec36eaaaba14fd7a445e32f8049e2d0; no voters:
I20260620 02:41:27.743669 22031 leader_election.cc:288] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:27.745057 22041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:27.746160 22041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [term 1 LEADER]: Becoming Leader. State: Replica: 8ec36eaaaba14fd7a445e32f8049e2d0, State: Running, Role: LEADER
I20260620 02:41:27.747156 22031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:27.747364 22041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36eaaaba14fd7a445e32f8049e2d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42885 } }
I20260620 02:41:27.757191 22043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ec36eaaaba14fd7a445e32f8049e2d0. Latest consensus state: current_term: 1 leader_uuid: "8ec36eaaaba14fd7a445e32f8049e2d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36eaaaba14fd7a445e32f8049e2d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42885 } } }
I20260620 02:41:27.757938 22043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:27.757202 22042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ec36eaaaba14fd7a445e32f8049e2d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36eaaaba14fd7a445e32f8049e2d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42885 } } }
I20260620 02:41:27.758198 22042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:27.759588 22055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:27.766088 22055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:27.779564 22055 catalog_manager.cc:1347] Generated new cluster ID: bd0354dce0c0434b87182d0efb509e47
I20260620 02:41:27.780107 22055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:27.799554 22055 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:27.802827 22055 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:27.815778 22055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ec36eaaaba14fd7a445e32f8049e2d0: Generated new TSK 0
I20260620 02:41:27.817116 22055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:27.835937 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21964
W20260620 02:41:27.837972 22026 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.93:47401
W20260620 02:41:27.839288 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 21964: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:27.839538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 21964
W20260620 02:41:27.857928 21949 connection.cc:535] client connection to 127.0.0.1:42885 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42885): Connection reset by peer (error 104)
2026-06-20T02:41:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:27 dist-test-slave-rpg6 krb5kdc[22062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:27 dist-test-slave-rpg6 krb5kdc[22062](info): set up 2 sockets
Jun 20 02:41:27 dist-test-slave-rpg6 krb5kdc[22062](info): commencing operation
krb5kdc: starting...
W20260620 02:41:29.057121 388 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:29 dist-test-slave-rpg6 krb5kdc[22062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:41:29.352072 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:29.731007 22075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:29.731349 22075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:29.731489 22075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:29.739661 22075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:29.739857 22075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:29.740109 22075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:29.740219 22075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:29.740303 22075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:29.751718 22075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:29.754127 22075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:29.755759 22075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:29.769982 22080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:29.771875 22083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:29.771896 22081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:29.772408 22075 server_base.cc:1034] running on GCE node
I20260620 02:41:29.773402 22075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:29.775130 22075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:29.776419 22075 hybrid_clock.cc:648] HybridClock initialized: now 1781923289776353 us; error 69 us; skew 500 ppm
I20260620 02:41:29.777024 22075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:29 dist-test-slave-rpg6 krb5kdc[22062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:29.781529 22075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:41:29.783313 22075 webserver.cc:460] Webserver started at http://127.0.0.1:41711/ using document root <none> and password file <none>
I20260620 02:41:29.784168 22075 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:29.784303 22075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:29.784574 22075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:29.788134 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "0a2e7ff405824c12a1e0416810d2da44"
format_stamp: "Formatted at 2026-06-20 02:41:29 on dist-test-slave-rpg6"
server_key: "3b199ef74a29d65ce6a66eeae263232c"
server_key_iv: "65877e063db2f413d379d888f605db75"
server_key_version: "encryptionkey@0"
I20260620 02:41:29.789011 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "0a2e7ff405824c12a1e0416810d2da44"
format_stamp: "Formatted at 2026-06-20 02:41:29 on dist-test-slave-rpg6"
server_key: "3b199ef74a29d65ce6a66eeae263232c"
server_key_iv: "65877e063db2f413d379d888f605db75"
server_key_version: "encryptionkey@0"
I20260620 02:41:29.796097 22075 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260620 02:41:29.800966 22090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:29.803539 22075 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:41:29.803902 22075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "0a2e7ff405824c12a1e0416810d2da44"
format_stamp: "Formatted at 2026-06-20 02:41:29 on dist-test-slave-rpg6"
server_key: "3b199ef74a29d65ce6a66eeae263232c"
server_key_iv: "65877e063db2f413d379d888f605db75"
server_key_version: "encryptionkey@0"
I20260620 02:41:29.804195 22075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:29.813014 22075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:29.818261 22075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:29.818634 22075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:29.841984 22075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40643
I20260620 02:41:29.842175 22141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40643 every 8 connection(s)
I20260620 02:41:29.843794 22075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:29.849013 22142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:29.852807 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 22075
I20260620 02:41:29.853036 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:29.853683 388 external_mini_cluster.cc:1428] Setting key 1133b4dd6003fc76cc8c44c0c8490906
I20260620 02:41:29.864108 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:29.856605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:40433 (local address 127.0.0.1:40643)
0620 02:41:29.857526 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:29.857539 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:29.857587 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:41:29.859881 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:29.859916 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:29.860333 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:29.860772 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:29.861911 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:29.861918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:29.861932 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:29.861947 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:29.862011 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:29.862035 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:29.862319 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:29.862476 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 02:41:29.862694 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":453,"threads_started":1}
I20260620 02:41:29.865157 22142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44: Bootstrap starting.
I20260620 02:41:29.868614 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:29.868958 22142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:29.870087 22142 log.cc:824] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:29.873890 22142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44: No bootstrap required, opened a new log
I20260620 02:41:29.876906 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:29.873503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:35069 (local address 127.0.0.1:40643)
0620 02:41:29.873865 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:29.873872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:29.874013 (+ 141us) server_negotiation.cc:407] Connection header received
0620 02:41:29.874273 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:29.874280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:29.874342 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:29.874465 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:29.875815 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:29.875821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:29.875929 (+ 108us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:29.875971 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:29.876056 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:29.876082 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:29.876257 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:29.876363 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 02:41:29.876473 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260620 02:41:29.883549 22142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e7ff405824c12a1e0416810d2da44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } }
I20260620 02:41:29.884099 22142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:29.884210 22142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2e7ff405824c12a1e0416810d2da44, State: Initialized, Role: FOLLOWER
I20260620 02:41:29.884874 22142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e7ff405824c12a1e0416810d2da44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } }
I20260620 02:41:29.885133 22142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:29.885291 22142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:29.885474 22142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:29.888324 22142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e7ff405824c12a1e0416810d2da44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } }
I20260620 02:41:29.889076 22142 leader_election.cc:302] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a2e7ff405824c12a1e0416810d2da44; no voters:
I20260620 02:41:29.889814 22142 leader_election.cc:288] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:29.890002 22152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:29.890851 22152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2e7ff405824c12a1e0416810d2da44, State: Running, Role: LEADER
I20260620 02:41:29.891441 22152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e7ff405824c12a1e0416810d2da44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } }
I20260620 02:41:29.892140 22142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:29.894451 22154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a2e7ff405824c12a1e0416810d2da44. Latest consensus state: current_term: 1 leader_uuid: "0a2e7ff405824c12a1e0416810d2da44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e7ff405824c12a1e0416810d2da44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } } }
I20260620 02:41:29.894945 22154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:29.895525 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a2e7ff405824c12a1e0416810d2da44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2e7ff405824c12a1e0416810d2da44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40643 } } }
I20260620 02:41:29.895879 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:29.898082 22158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:29.906550 22158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:29.917904 22158 catalog_manager.cc:1347] Generated new cluster ID: fe9417fac4b6450591e607f8c6d27213
I20260620 02:41:29.918054 22158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:29.938205 22158 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:29.938752 22158 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:29.947103 22158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a2e7ff405824c12a1e0416810d2da44: Generated new TSK 0
I20260620 02:41:29.948395 22158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:29.960496 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22075
W20260620 02:41:29.962208 22137 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.93:40433
W20260620 02:41:29.963174 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22075: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:29.963359 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 22075
2026-06-20T02:41:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2124 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:30 dist-test-slave-rpg6 krb5kdc[22174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:30 dist-test-slave-rpg6 krb5kdc[22174](info): set up 2 sockets
Jun 20 02:41:30 dist-test-slave-rpg6 krb5kdc[22174](info): commencing operation
krb5kdc: starting...
W20260620 02:41:31.152617 388 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:31 dist-test-slave-rpg6 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:41:31.454077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44717
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:31.832547 22187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:31.832971 22187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:31.833132 22187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:31.840894 22187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:31.841044 22187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:31.841090 22187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:31.841185 22187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:31.841262 22187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:31.851861 22187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:31.853749 22187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:31.855456 22187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:31.869990 22192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:31.870061 22193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:31.870004 22195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:31.870304 22187 server_base.cc:1034] running on GCE node
I20260620 02:41:31.872056 22187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:31.874229 22187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:31.875733 22187 hybrid_clock.cc:648] HybridClock initialized: now 1781923291875608 us; error 113 us; skew 500 ppm
I20260620 02:41:31.876448 22187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:31 dist-test-slave-rpg6 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:31.885572 22187 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:41:31.887399 22187 webserver.cc:460] Webserver started at http://127.0.97.62:42459/ using document root <none> and password file <none>
I20260620 02:41:31.888286 22187 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:31.888453 22187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:31.888793 22187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:31.892220 22187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "63c143e8049d463cb12dbdeb861b5595"
format_stamp: "Formatted at 2026-06-20 02:41:31 on dist-test-slave-rpg6"
server_key: "d2df8c073e72c2368bd1298d2ac1aefa"
server_key_iv: "9c8d0c2cd93b693ca1c3cf5fb3e0c3dd"
server_key_version: "encryptionkey@0"
I20260620 02:41:31.893121 22187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "63c143e8049d463cb12dbdeb861b5595"
format_stamp: "Formatted at 2026-06-20 02:41:31 on dist-test-slave-rpg6"
server_key: "d2df8c073e72c2368bd1298d2ac1aefa"
server_key_iv: "9c8d0c2cd93b693ca1c3cf5fb3e0c3dd"
server_key_version: "encryptionkey@0"
I20260620 02:41:31.900146 22187 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 02:41:31.905543 22202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:31.908051 22187 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:41:31.908421 22187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "63c143e8049d463cb12dbdeb861b5595"
format_stamp: "Formatted at 2026-06-20 02:41:31 on dist-test-slave-rpg6"
server_key: "d2df8c073e72c2368bd1298d2ac1aefa"
server_key_iv: "9c8d0c2cd93b693ca1c3cf5fb3e0c3dd"
server_key_version: "encryptionkey@0"
I20260620 02:41:31.908746 22187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:31.936590 22187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:31.942544 22187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:31.942831 22187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:31.964526 22187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44717
I20260620 02:41:31.964556 22253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44717 every 8 connection(s)
I20260620 02:41:31.966094 22187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:31.970603 22254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:31.976310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 22187
I20260620 02:41:31.976521 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:31.977025 388 external_mini_cluster.cc:1428] Setting key f8f5a62d1458e81ca1fb03a700eb84d0
I20260620 02:41:31.983160 22254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595: Bootstrap starting.
I20260620 02:41:31.986845 22254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:31.988191 22254 log.cc:824] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:31.991496 22254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595: No bootstrap required, opened a new log
Jun 20 02:41:31 dist-test-slave-rpg6 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:41:32.002840 22254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c143e8049d463cb12dbdeb861b5595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44717 } }
I20260620 02:41:32.003466 22254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:32.003660 22254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c143e8049d463cb12dbdeb861b5595, State: Initialized, Role: FOLLOWER
I20260620 02:41:32.004479 22254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c143e8049d463cb12dbdeb861b5595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44717 } }
I20260620 02:41:32.004810 22254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:32.004992 22254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:32.004969 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:31.979244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50063 (local address 127.0.97.62:44717)
0620 02:41:31.980334 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:31.980352 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:31.980408 (+ 56us) server_negotiation.cc:407] Connection header received
0620 02:41:31.982468 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:31.982497 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:31.983026 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:31.983505 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:31.985321 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:31.986847 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:31.988575 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:31.989440 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:31.989693 (+ 253us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:31.994523 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:31.994533 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:31.994551 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:31.994608 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:31.999269 (+ 4661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:32.000765 (+ 1496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:32.000777 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:32.000785 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:32.001003 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:32.001551 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:32.001555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:32.001557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:32.002235 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:32.002602 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:32.003130 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 02:41:32.003567 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":546,"threads_started":1}
I20260620 02:41:32.005280 22254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:32.008999 22254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c143e8049d463cb12dbdeb861b5595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44717 } }
I20260620 02:41:32.009734 22254 leader_election.cc:302] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63c143e8049d463cb12dbdeb861b5595; no voters:
I20260620 02:41:32.010608 22254 leader_election.cc:288] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:32.010743 22259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:32.011528 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:32.012719 22259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [term 1 LEADER]: Becoming Leader. State: Replica: 63c143e8049d463cb12dbdeb861b5595, State: Running, Role: LEADER
I20260620 02:41:32.013522 22259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c143e8049d463cb12dbdeb861b5595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44717 } }
I20260620 02:41:32.013998 22254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:32.018330 22260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63c143e8049d463cb12dbdeb861b5595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c143e8049d463cb12dbdeb861b5595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44717 } } }
I20260620 02:41:32.018956 22260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:32.019769 22261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63c143e8049d463cb12dbdeb861b5595. Latest consensus state: current_term: 1 leader_uuid: "63c143e8049d463cb12dbdeb861b5595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c143e8049d463cb12dbdeb861b5595" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44717 } } }
I20260620 02:41:32.020251 22261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:32.029794 22267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:32.036314 22267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:32.059428 22267 catalog_manager.cc:1347] Generated new cluster ID: f208fa1acbec42189a464ce9f20077cd
I20260620 02:41:32.059645 22267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:32.063164 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:32.029676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32849 (local address 127.0.97.62:44717)
0620 02:41:32.029874 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:32.029882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:32.029909 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:41:32.030367 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:32.030374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:32.030466 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:32.030623 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:32.033611 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:32.035537 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:32.039038 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:32.040172 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:32.040346 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:32.049032 (+ 8686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:32.049040 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:32.049046 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:32.049104 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:32.055111 (+ 6007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:32.057184 (+ 2073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:32.057196 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:32.057203 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:32.057493 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:32.058338 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:32.058347 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:32.058353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:32.059029 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:32.059537 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:32.062425 (+ 2888us) server_negotiation.cc:299] Negotiation successful
0620 02:41:32.062732 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:41:32.080924 22267 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:32.083256 22267 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:32.093711 22267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63c143e8049d463cb12dbdeb861b5595: Generated new TSK 0
I20260620 02:41:32.095031 22267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:32.103039 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22187
I20260620 02:41:32.104979 22249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:32.201300 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 22187
W20260620 02:41:32.220489 22172 connection.cc:535] client connection to 127.0.97.62:44717 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:44717): Connection reset by peer (error 104)
2026-06-20T02:41:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2237 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:32 dist-test-slave-rpg6 krb5kdc[22287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:32 dist-test-slave-rpg6 krb5kdc[22287](info): set up 2 sockets
Jun 20 02:41:32 dist-test-slave-rpg6 krb5kdc[22287](info): commencing operation
krb5kdc: starting...
W20260620 02:41:33.364629 388 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.004s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:33 dist-test-slave-rpg6 krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:41:33.665273 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35655
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:34.047164 22300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:34.047506 22300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:34.047684 22300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:34.056083 22300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:34.056226 22300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:34.056313 22300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:34.056581 22300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:34.056671 22300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:34.067569 22300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:34.069087 22300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:34.070571 22300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:34.084797 22305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:34.084690 22308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:34.085567 22306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:34.085644 22300 server_base.cc:1034] running on GCE node
I20260620 02:41:34.086764 22300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:34.088691 22300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:34.090204 22300 hybrid_clock.cc:648] HybridClock initialized: now 1781923294090081 us; error 122 us; skew 500 ppm
I20260620 02:41:34.090658 22300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:34 dist-test-slave-rpg6 krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:34.095252 22300 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:41:34.097386 22300 webserver.cc:460] Webserver started at http://127.0.97.62:37583/ using document root <none> and password file <none>
I20260620 02:41:34.098215 22300 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:34.098387 22300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:34.098743 22300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:34.102589 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "1c22e9e5a4cc46f6a41b3ce72390e240"
format_stamp: "Formatted at 2026-06-20 02:41:34 on dist-test-slave-rpg6"
server_key: "1da4a15b633d5584f427b33967be7643"
server_key_iv: "a11d6c3afa2eb95a82078559f9c268b1"
server_key_version: "encryptionkey@0"
I20260620 02:41:34.103646 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "1c22e9e5a4cc46f6a41b3ce72390e240"
format_stamp: "Formatted at 2026-06-20 02:41:34 on dist-test-slave-rpg6"
server_key: "1da4a15b633d5584f427b33967be7643"
server_key_iv: "a11d6c3afa2eb95a82078559f9c268b1"
server_key_version: "encryptionkey@0"
I20260620 02:41:34.110926 22300 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 02:41:34.115808 22315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:34.118260 22300 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260620 02:41:34.118564 22300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "1c22e9e5a4cc46f6a41b3ce72390e240"
format_stamp: "Formatted at 2026-06-20 02:41:34 on dist-test-slave-rpg6"
server_key: "1da4a15b633d5584f427b33967be7643"
server_key_iv: "a11d6c3afa2eb95a82078559f9c268b1"
server_key_version: "encryptionkey@0"
I20260620 02:41:34.118839 22300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:34.140520 22300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:34.146169 22300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:34.146533 22300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:34.168676 22300 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35655
I20260620 02:41:34.168913 22366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35655 every 8 connection(s)
I20260620 02:41:34.170373 22300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:34.176132 22367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:34.177155 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 22300
I20260620 02:41:34.177414 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:34.177942 388 external_mini_cluster.cc:1428] Setting key 378e8b7149177faede0d99134d945c69
I20260620 02:41:34.190608 22367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240: Bootstrap starting.
I20260620 02:41:34.192865 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:34.180422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53343 (local address 127.0.97.62:35655)
0620 02:41:34.181371 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:34.181383 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:34.181427 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:41:34.183318 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:34.183345 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:34.183913 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:34.184411 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:34.185019 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:34.186358 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:34.187761 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:34.188622 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:34.188841 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:34.190259 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:34.190267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:34.190283 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:34.190963 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:34.191059 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:34.191102 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:34.191288 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:34.191584 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 02:41:34.191771 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":427,"threads_started":1}
I20260620 02:41:34.194304 22367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:34.195622 22367 log.cc:824] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:34.198753 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:34.200954 22367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240: No bootstrap required, opened a new log
I20260620 02:41:34.210716 22367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c22e9e5a4cc46f6a41b3ce72390e240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35655 } }
I20260620 02:41:34.211187 22367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:34.211381 22367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c22e9e5a4cc46f6a41b3ce72390e240, State: Initialized, Role: FOLLOWER
I20260620 02:41:34.212260 22367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c22e9e5a4cc46f6a41b3ce72390e240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35655 } }
I20260620 02:41:34.212599 22367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:34.212776 22367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:34.212807 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:34.203534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54893 (local address 127.0.97.62:35655)
0620 02:41:34.203868 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:34.203878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:34.204178 (+ 300us) server_negotiation.cc:407] Connection header received
0620 02:41:34.204457 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:34.204462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:34.204537 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:34.204668 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:34.205251 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:34.206105 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:34.207693 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:34.208777 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:34.209222 (+ 445us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:34.211432 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:34.211442 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:34.211448 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:34.211477 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:34.211560 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:34.211621 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:34.211784 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:34.212245 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 02:41:34.212351 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:41:34.213013 22367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:34.215798 22367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c22e9e5a4cc46f6a41b3ce72390e240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35655 } }
I20260620 02:41:34.216567 22367 leader_election.cc:302] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c22e9e5a4cc46f6a41b3ce72390e240; no voters:
I20260620 02:41:34.217183 22367 leader_election.cc:288] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:34.217407 22377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:34.218397 22377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [term 1 LEADER]: Becoming Leader. State: Replica: 1c22e9e5a4cc46f6a41b3ce72390e240, State: Running, Role: LEADER
I20260620 02:41:34.219156 22377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c22e9e5a4cc46f6a41b3ce72390e240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35655 } }
I20260620 02:41:34.220053 22367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:34.223189 22379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c22e9e5a4cc46f6a41b3ce72390e240. Latest consensus state: current_term: 1 leader_uuid: "1c22e9e5a4cc46f6a41b3ce72390e240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c22e9e5a4cc46f6a41b3ce72390e240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35655 } } }
I20260620 02:41:34.223726 22379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:34.223219 22378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c22e9e5a4cc46f6a41b3ce72390e240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c22e9e5a4cc46f6a41b3ce72390e240" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35655 } } }
I20260620 02:41:34.224783 22378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:34.228549 22385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:34.235433 22385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:34.248009 22385 catalog_manager.cc:1347] Generated new cluster ID: 9dab2cac78984049b2e252970c49a77a
I20260620 02:41:34.248245 22385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:34.262563 22385 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:34.265130 22385 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:34.273670 22385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c22e9e5a4cc46f6a41b3ce72390e240: Generated new TSK 0
I20260620 02:41:34.274724 22385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:34.290138 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22300
W20260620 02:41:34.292474 22362 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53343
W20260620 02:41:34.294459 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22300: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:34.294659 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 22300
W20260620 02:41:34.314438 22285 connection.cc:535] client connection to 127.0.97.62:35655 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:35655): Connection reset by peer (error 104)
2026-06-20T02:41:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2093 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:34 dist-test-slave-rpg6 krb5kdc[22398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:34 dist-test-slave-rpg6 krb5kdc[22398](info): set up 2 sockets
Jun 20 02:41:34 dist-test-slave-rpg6 krb5kdc[22398](info): commencing operation
krb5kdc: starting...
W20260620 02:41:35.480819 388 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:35 dist-test-slave-rpg6 krb5kdc[22398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:41:35.780913 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41333
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:36.165861 22411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:36.166216 22411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:36.166414 22411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:36.174854 22411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:36.175016 22411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:36.175125 22411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:36.175252 22411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:36.175323 22411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:36.186030 22411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41333
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:36.187562 22411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:36.189092 22411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:36.205008 22416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:36.204985 22419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:36.204963 22417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:36.206892 22411 server_base.cc:1034] running on GCE node
I20260620 02:41:36.208081 22411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:36.210209 22411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:36.211972 22411 hybrid_clock.cc:648] HybridClock initialized: now 1781923296211851 us; error 114 us; skew 500 ppm
I20260620 02:41:36.212476 22411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:36 dist-test-slave-rpg6 krb5kdc[22398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:36.217068 22411 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:41:36.220155 22411 webserver.cc:460] Webserver started at http://127.0.97.62:37027/ using document root <none> and password file <none>
I20260620 02:41:36.221366 22411 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:36.221606 22411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:36.222079 22411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:36.226662 22411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "306a282e57ce479ab20e1d728a1824e1"
format_stamp: "Formatted at 2026-06-20 02:41:36 on dist-test-slave-rpg6"
server_key: "4898a2257ac704e1d4b72e9d0fd250e2"
server_key_iv: "8a6cd63101f96ca6e4406459330818e6"
server_key_version: "encryptionkey@0"
I20260620 02:41:36.227680 22411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "306a282e57ce479ab20e1d728a1824e1"
format_stamp: "Formatted at 2026-06-20 02:41:36 on dist-test-slave-rpg6"
server_key: "4898a2257ac704e1d4b72e9d0fd250e2"
server_key_iv: "8a6cd63101f96ca6e4406459330818e6"
server_key_version: "encryptionkey@0"
I20260620 02:41:36.234845 22411 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 02:41:36.241216 22426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:36.243201 22411 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260620 02:41:36.243481 22411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "306a282e57ce479ab20e1d728a1824e1"
format_stamp: "Formatted at 2026-06-20 02:41:36 on dist-test-slave-rpg6"
server_key: "4898a2257ac704e1d4b72e9d0fd250e2"
server_key_iv: "8a6cd63101f96ca6e4406459330818e6"
server_key_version: "encryptionkey@0"
I20260620 02:41:36.243753 22411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:36.250941 22411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:36.257462 22411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:36.257849 22411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:36.283228 22477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41333 every 8 connection(s)
I20260620 02:41:36.283560 22411 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41333
I20260620 02:41:36.285497 22411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:36.291306 22478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:36.292605 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 22411
I20260620 02:41:36.292846 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:36.293548 388 external_mini_cluster.cc:1428] Setting key 62b2880f50ed2ecbfe9d04b725f87ac8
I20260620 02:41:36.307595 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:36.296802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57563 (local address 127.0.97.62:41333)
0620 02:41:36.298290 (+ 1488us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:36.298317 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:36.300946 (+ 2629us) server_negotiation.cc:407] Connection header received
0620 02:41:36.303046 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:36.303074 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:36.303491 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:36.303975 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:36.305213 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:36.305220 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:36.305240 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:36.305267 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:36.305345 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:36.305374 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:36.305603 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:36.305712 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 02:41:36.306122 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":686,"threads_started":1}
I20260620 02:41:36.308990 22478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1: Bootstrap starting.
I20260620 02:41:36.313378 22478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:36.314993 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:36.314982 22478 log.cc:824] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:36.320600 22478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1: No bootstrap required, opened a new log
I20260620 02:41:36.326152 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:36.321520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48115 (local address 127.0.97.62:41333)
0620 02:41:36.321814 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:36.321824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:36.322485 (+ 661us) server_negotiation.cc:407] Connection header received
0620 02:41:36.322654 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:36.322660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:36.322756 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:36.322915 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:36.324394 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:36.324408 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:36.324415 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:36.324450 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:36.324722 (+ 272us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:36.324765 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:36.324948 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:36.325227 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 02:41:36.325525 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:41:36.333060 22478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a282e57ce479ab20e1d728a1824e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41333 } }
I20260620 02:41:36.333555 22478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:36.333674 22478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306a282e57ce479ab20e1d728a1824e1, State: Initialized, Role: FOLLOWER
I20260620 02:41:36.334399 22478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a282e57ce479ab20e1d728a1824e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41333 } }
I20260620 02:41:36.334641 22478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:36.334789 22478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:36.334985 22478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:36.337631 22478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a282e57ce479ab20e1d728a1824e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41333 } }
I20260620 02:41:36.338256 22478 leader_election.cc:302] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 306a282e57ce479ab20e1d728a1824e1; no voters:
I20260620 02:41:36.338955 22478 leader_election.cc:288] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:36.339187 22488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:36.341414 22488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [term 1 LEADER]: Becoming Leader. State: Replica: 306a282e57ce479ab20e1d728a1824e1, State: Running, Role: LEADER
I20260620 02:41:36.342126 22488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a282e57ce479ab20e1d728a1824e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41333 } }
I20260620 02:41:36.342360 22478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:36.345808 22490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 306a282e57ce479ab20e1d728a1824e1. Latest consensus state: current_term: 1 leader_uuid: "306a282e57ce479ab20e1d728a1824e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a282e57ce479ab20e1d728a1824e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41333 } } }
I20260620 02:41:36.345890 22489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "306a282e57ce479ab20e1d728a1824e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a282e57ce479ab20e1d728a1824e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41333 } } }
I20260620 02:41:36.346350 22489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:36.346350 22490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:36.349704 22496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:36.356040 22496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:36.367455 22496 catalog_manager.cc:1347] Generated new cluster ID: 3d90822655c14b4cba83d52912ea9f58
I20260620 02:41:36.367662 22496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:36.383471 22496 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:36.384196 22496 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:36.395541 22496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 306a282e57ce479ab20e1d728a1824e1: Generated new TSK 0
I20260620 02:41:36.396908 22496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:36.413851 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22411
W20260620 02:41:36.415448 22473 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57563
W20260620 02:41:36.416446 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22411: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:36.416630 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 22411
2026-06-20T02:41:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:36 dist-test-slave-rpg6 krb5kdc[22509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:36 dist-test-slave-rpg6 krb5kdc[22509](info): set up 2 sockets
Jun 20 02:41:36 dist-test-slave-rpg6 krb5kdc[22509](info): commencing operation
krb5kdc: starting...
W20260620 02:41:37.597265 388 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.001s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:37 dist-test-slave-rpg6 krb5kdc[22509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:41:37.905158 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34787
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:38.300403 22522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:38.300786 22522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:38.300908 22522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:38.310016 22522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:38.310173 22522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:38.310230 22522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:38.310325 22522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:38.310375 22522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:38.323938 22522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:38.325471 22522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:38.326871 22522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:38.343357 22527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:38.344025 22528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:38.344553 22522 server_base.cc:1034] running on GCE node
W20260620 02:41:38.344753 22530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:38.345813 22522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:38.347654 22522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:38.349074 22522 hybrid_clock.cc:648] HybridClock initialized: now 1781923298349004 us; error 113 us; skew 500 ppm
I20260620 02:41:38.349821 22522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:38 dist-test-slave-rpg6 krb5kdc[22509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:38.358613 22522 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:41:38.361370 22522 webserver.cc:460] Webserver started at http://127.0.97.62:46027/ using document root <none> and password file <none>
I20260620 02:41:38.362396 22522 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:38.362586 22522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:38.362946 22522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:38.366638 22522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "9b02419568b04ed9bcac0bee691385c6"
format_stamp: "Formatted at 2026-06-20 02:41:38 on dist-test-slave-rpg6"
server_key: "d0b8130c5f55fe9052eece797301c992"
server_key_iv: "aaec65943180a26329b54ac6aca2b6b7"
server_key_version: "encryptionkey@0"
I20260620 02:41:38.367697 22522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "9b02419568b04ed9bcac0bee691385c6"
format_stamp: "Formatted at 2026-06-20 02:41:38 on dist-test-slave-rpg6"
server_key: "d0b8130c5f55fe9052eece797301c992"
server_key_iv: "aaec65943180a26329b54ac6aca2b6b7"
server_key_version: "encryptionkey@0"
I20260620 02:41:38.377345 22522 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260620 02:41:38.383325 22537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:38.385509 22522 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:41:38.385926 22522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "9b02419568b04ed9bcac0bee691385c6"
format_stamp: "Formatted at 2026-06-20 02:41:38 on dist-test-slave-rpg6"
server_key: "d0b8130c5f55fe9052eece797301c992"
server_key_iv: "aaec65943180a26329b54ac6aca2b6b7"
server_key_version: "encryptionkey@0"
I20260620 02:41:38.386236 22522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:38.416916 22522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:38.422293 22522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:38.422596 22522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:38.446666 22522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34787
I20260620 02:41:38.446904 22588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34787 every 8 connection(s)
I20260620 02:41:38.448485 22522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:38.452836 22589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:38.458017 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 22522
I20260620 02:41:38.458228 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:38.458719 388 external_mini_cluster.cc:1428] Setting key fa923926757fd4ba78c4e453592be3b8
I20260620 02:41:38.465854 22589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6: Bootstrap starting.
I20260620 02:41:38.470274 22589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:38.471782 22589 log.cc:824] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:38.476814 22589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6: No bootstrap required, opened a new log
Jun 20 02:41:38 dist-test-slave-rpg6 krb5kdc[22509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:41:38.486515 22589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b02419568b04ed9bcac0bee691385c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34787 } }
I20260620 02:41:38.487083 22589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:38.487252 22589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b02419568b04ed9bcac0bee691385c6, State: Initialized, Role: FOLLOWER
I20260620 02:41:38.488147 22589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b02419568b04ed9bcac0bee691385c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34787 } }
I20260620 02:41:38.488461 22589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:38.488634 22589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:38.488866 22589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:38.493552 22589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b02419568b04ed9bcac0bee691385c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34787 } }
I20260620 02:41:38.494328 22589 leader_election.cc:302] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b02419568b04ed9bcac0bee691385c6; no voters:
I20260620 02:41:38.495203 22589 leader_election.cc:288] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:38.495362 22594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:38.496066 22594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [term 1 LEADER]: Becoming Leader. State: Replica: 9b02419568b04ed9bcac0bee691385c6, State: Running, Role: LEADER
I20260620 02:41:38.496765 22594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b02419568b04ed9bcac0bee691385c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34787 } }
I20260620 02:41:38.497044 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:38.460968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42467 (local address 127.0.97.62:34787)
0620 02:41:38.462533 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:38.462559 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:38.462639 (+ 80us) server_negotiation.cc:407] Connection header received
0620 02:41:38.465181 (+ 2542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:38.465219 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:38.465899 (+ 680us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:38.466478 (+ 579us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:38.468846 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:38.470355 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:38.472549 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:38.474128 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:38.474301 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:38.484997 (+ 10696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:38.485008 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:38.485034 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:38.485166 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:38.489181 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:38.490782 (+ 1601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:38.490796 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:38.490812 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:38.491189 (+ 377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:38.491878 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:38.491884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:38.491888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:38.493039 (+ 1151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:38.493426 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:38.494289 (+ 863us) server_negotiation.cc:299] Negotiation successful
0620 02:41:38.494935 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":470,"threads_started":1}
I20260620 02:41:38.498807 22594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b02419568b04ed9bcac0bee691385c6. Latest consensus state: current_term: 1 leader_uuid: "9b02419568b04ed9bcac0bee691385c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b02419568b04ed9bcac0bee691385c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34787 } } }
I20260620 02:41:38.499472 22594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:38.499634 22596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b02419568b04ed9bcac0bee691385c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b02419568b04ed9bcac0bee691385c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34787 } } }
I20260620 02:41:38.500062 22596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:38.501425 22589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:38.503202 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:38.504586 22598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:38.510726 22598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:38.529465 22598 catalog_manager.cc:1347] Generated new cluster ID: 4dd1531949ce4a029b74e44cc8546f05
I20260620 02:41:38.529618 22598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:38.543272 22598 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:38.545023 22598 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:38.548979 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:38.524401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58497 (local address 127.0.97.62:34787)
0620 02:41:38.524691 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:38.524700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:38.526192 (+ 1492us) server_negotiation.cc:407] Connection header received
0620 02:41:38.526596 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:38.526603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:38.526693 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:38.526821 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:38.530859 (+ 4038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:38.532491 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:38.535241 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:38.536153 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:38.536345 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:38.541441 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:38.541449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:38.541454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:38.541500 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:38.545259 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:38.546337 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:38.546348 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:38.546352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:38.546537 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:38.547032 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:38.547037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:38.547040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:38.547436 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:38.547702 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:38.548067 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 02:41:38.548358 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 02:41:38.557670 22598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b02419568b04ed9bcac0bee691385c6: Generated new TSK 0
I20260620 02:41:38.558925 22598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:38.568105 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22522
I20260620 02:41:38.570034 22584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:38.670783 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 22522
2026-06-20T02:41:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2260 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:38 dist-test-slave-rpg6 krb5kdc[22621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:38 dist-test-slave-rpg6 krb5kdc[22621](info): set up 2 sockets
Jun 20 02:41:38 dist-test-slave-rpg6 krb5kdc[22621](info): commencing operation
krb5kdc: starting...
W20260620 02:41:39.877179 388 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:40 dist-test-slave-rpg6 krb5kdc[22621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:41:40.193972 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34103
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:40.566912 22634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:40.567313 22634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:40.567488 22634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:40.575311 22634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:40.575513 22634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:40.575706 22634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:40.575861 22634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:40.575938 22634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:40.586587 22634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:40.588104 22634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:40.589505 22634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:40.602831 22640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:40.602911 22642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:40.602856 22639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:40.603562 22634 server_base.cc:1034] running on GCE node
I20260620 02:41:40.604552 22634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:40.606338 22634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:40.607668 22634 hybrid_clock.cc:648] HybridClock initialized: now 1781923300607604 us; error 59 us; skew 500 ppm
I20260620 02:41:40.608151 22634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:40 dist-test-slave-rpg6 krb5kdc[22621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:40.612107 22634 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:41:40.614385 22634 webserver.cc:460] Webserver started at http://127.0.97.62:44147/ using document root <none> and password file <none>
I20260620 02:41:40.615197 22634 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:40.615365 22634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:40.615700 22634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:40.619196 22634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "dd44922052e34ae7a3f9bd4a42b1fd09"
format_stamp: "Formatted at 2026-06-20 02:41:40 on dist-test-slave-rpg6"
server_key: "f0f98494d973bd32301b6f663d1b94db"
server_key_iv: "eba9721144681a7e1793f980d329b4fb"
server_key_version: "encryptionkey@0"
I20260620 02:41:40.620198 22634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "dd44922052e34ae7a3f9bd4a42b1fd09"
format_stamp: "Formatted at 2026-06-20 02:41:40 on dist-test-slave-rpg6"
server_key: "f0f98494d973bd32301b6f663d1b94db"
server_key_iv: "eba9721144681a7e1793f980d329b4fb"
server_key_version: "encryptionkey@0"
I20260620 02:41:40.626291 22634 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:41:40.631388 22649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:40.633114 22634 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 02:41:40.633345 22634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "dd44922052e34ae7a3f9bd4a42b1fd09"
format_stamp: "Formatted at 2026-06-20 02:41:40 on dist-test-slave-rpg6"
server_key: "f0f98494d973bd32301b6f663d1b94db"
server_key_iv: "eba9721144681a7e1793f980d329b4fb"
server_key_version: "encryptionkey@0"
I20260620 02:41:40.633592 22634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:40.659942 22634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:40.664783 22634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:40.665127 22634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:40.687994 22634 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34103
I20260620 02:41:40.688158 22700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34103 every 8 connection(s)
I20260620 02:41:40.689507 22634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:40.693715 22701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:40.698637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 22634
I20260620 02:41:40.698879 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:40.699397 388 external_mini_cluster.cc:1428] Setting key dad3aebef35997181a31454c1731bef1
I20260620 02:41:40.706442 22701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09: Bootstrap starting.
I20260620 02:41:40.710644 22701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:40.712092 22701 log.cc:824] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:40.716473 22701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09: No bootstrap required, opened a new log
I20260620 02:41:40.716773 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:40.701972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35551 (local address 127.0.97.62:34103)
0620 02:41:40.703333 (+ 1361us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:40.703351 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:40.703400 (+ 49us) server_negotiation.cc:407] Connection header received
0620 02:41:40.705668 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:40.705699 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:40.706199 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:40.706685 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:40.707363 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:40.708986 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:40.711028 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:40.711929 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:40.712112 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:40.713534 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:40.713547 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:40.713573 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:40.714529 (+ 956us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:40.714618 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:40.714649 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:40.714882 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:40.715180 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 02:41:40.715518 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":761,"threads_started":1}
I20260620 02:41:40.721539 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:40.728543 22701 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd44922052e34ae7a3f9bd4a42b1fd09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:41:40.729171 22701 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:40.729358 22701 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd44922052e34ae7a3f9bd4a42b1fd09, State: Initialized, Role: FOLLOWER
I20260620 02:41:40.730140 22701 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd44922052e34ae7a3f9bd4a42b1fd09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:41:40.730396 22701 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:40.730583 22701 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:40.730875 22701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:40.733871 22701 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd44922052e34ae7a3f9bd4a42b1fd09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:41:40.734618 22701 leader_election.cc:302] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd44922052e34ae7a3f9bd4a42b1fd09; no voters:
I20260620 02:41:40.735687 22701 leader_election.cc:288] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:40.735924 22711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:40.736997 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:40.728040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35123 (local address 127.0.97.62:34103)
0620 02:41:40.728265 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:40.728272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:40.728717 (+ 445us) server_negotiation.cc:407] Connection header received
0620 02:41:40.728883 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:40.728890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:40.728985 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:40.729110 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:40.730505 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:40.731734 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:40.733378 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:40.734096 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:40.734240 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:40.735997 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:40.736006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:40.736015 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:40.736048 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:40.736115 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:40.736143 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:40.736274 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:40.736493 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 02:41:40.736624 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:41:40.738114 22711 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [term 1 LEADER]: Becoming Leader. State: Replica: dd44922052e34ae7a3f9bd4a42b1fd09, State: Running, Role: LEADER
I20260620 02:41:40.739405 22711 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd44922052e34ae7a3f9bd4a42b1fd09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:41:40.739738 22701 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:40.742823 22712 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd44922052e34ae7a3f9bd4a42b1fd09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd44922052e34ae7a3f9bd4a42b1fd09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } } }
I20260620 02:41:40.743793 22712 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:40.742895 22713 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd44922052e34ae7a3f9bd4a42b1fd09. Latest consensus state: current_term: 1 leader_uuid: "dd44922052e34ae7a3f9bd4a42b1fd09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd44922052e34ae7a3f9bd4a42b1fd09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } } }
I20260620 02:41:40.744491 22713 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:40.748593 22717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:40.755564 22717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:40.768030 22717 catalog_manager.cc:1347] Generated new cluster ID: 51377337f47448c8bc429bad73279136
I20260620 02:41:40.768179 22717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:40.788877 22717 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:40.790637 22717 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:40.801441 22717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd44922052e34ae7a3f9bd4a42b1fd09: Generated new TSK 0
I20260620 02:41:40.802690 22717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:40.810969 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22634
W20260620 02:41:40.812649 22696 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35551
W20260620 02:41:40.813681 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22634: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:40.813875 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 22634
2026-06-20T02:41:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2134 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:40 dist-test-slave-rpg6 krb5kdc[22732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:40 dist-test-slave-rpg6 krb5kdc[22732](info): set up 2 sockets
Jun 20 02:41:40 dist-test-slave-rpg6 krb5kdc[22732](info): commencing operation
krb5kdc: starting...
W20260620 02:41:41.973129 388 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:42 dist-test-slave-rpg6 krb5kdc[22732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:41:42.294746 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35335
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:42.670730 22745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:42.671137 22745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:42.671317 22745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:42.680310 22745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:42.680532 22745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:42.680653 22745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:42.680776 22745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:42.680867 22745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:42.691542 22745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:42.693090 22745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:42.694497 22745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:42.708719 22750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:42.708738 22753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:42.708770 22751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:42.710084 22745 server_base.cc:1034] running on GCE node
I20260620 02:41:42.711042 22745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:42.712934 22745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:42.714215 22745 hybrid_clock.cc:648] HybridClock initialized: now 1781923302714158 us; error 84 us; skew 500 ppm
I20260620 02:41:42.714622 22745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:42 dist-test-slave-rpg6 krb5kdc[22732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:42.718235 22745 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:41:42.719962 22745 webserver.cc:460] Webserver started at http://127.0.97.62:34171/ using document root <none> and password file <none>
I20260620 02:41:42.720695 22745 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:42.720808 22745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:42.721081 22745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:42.724494 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "55df05240e1044b195e076278d607e72"
format_stamp: "Formatted at 2026-06-20 02:41:42 on dist-test-slave-rpg6"
server_key: "4b07c1c26dfafc8e1e5257905342dab7"
server_key_iv: "1ee0b696eaff97449de3745eca7ea55c"
server_key_version: "encryptionkey@0"
I20260620 02:41:42.725361 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "55df05240e1044b195e076278d607e72"
format_stamp: "Formatted at 2026-06-20 02:41:42 on dist-test-slave-rpg6"
server_key: "4b07c1c26dfafc8e1e5257905342dab7"
server_key_iv: "1ee0b696eaff97449de3745eca7ea55c"
server_key_version: "encryptionkey@0"
I20260620 02:41:42.731276 22745 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:41:42.736110 22760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:42.738646 22745 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:41:42.738855 22745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "55df05240e1044b195e076278d607e72"
format_stamp: "Formatted at 2026-06-20 02:41:42 on dist-test-slave-rpg6"
server_key: "4b07c1c26dfafc8e1e5257905342dab7"
server_key_iv: "1ee0b696eaff97449de3745eca7ea55c"
server_key_version: "encryptionkey@0"
I20260620 02:41:42.739084 22745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:42.746069 22745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:42.751226 22745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:42.751526 22745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:42.774569 22745 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35335
I20260620 02:41:42.774731 22811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35335 every 8 connection(s)
I20260620 02:41:42.776312 22745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:42.779182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 22745
I20260620 02:41:42.779394 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:42.779942 388 external_mini_cluster.cc:1428] Setting key 612debe847d0d6a434787dba7968f09d
I20260620 02:41:42.780810 22812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:42.791062 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:42.783052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49899 (local address 127.0.97.62:35335)
0620 02:41:42.784449 (+ 1397us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:42.784471 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:42.784544 (+ 73us) server_negotiation.cc:407] Connection header received
0620 02:41:42.786355 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:42.786394 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:42.786982 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:42.787555 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:42.788849 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:42.788859 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:42.788879 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:42.788908 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:42.788999 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:42.789035 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:42.789295 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:42.789459 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 02:41:42.789695 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1057,"thread_start_us":742,"threads_started":1}
I20260620 02:41:42.797145 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:42.797857 22812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72: Bootstrap starting.
I20260620 02:41:42.802589 22812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:42.804199 22812 log.cc:824] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:42.806764 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:42.803260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56363 (local address 127.0.97.62:35335)
0620 02:41:42.803446 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:42.803452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:42.803877 (+ 425us) server_negotiation.cc:407] Connection header received
0620 02:41:42.804087 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:42.804092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:42.804167 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:42.804321 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:42.805924 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:42.805930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:42.805934 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:41:42.805955 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:42.806010 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:41:42.806030 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:42.806188 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:42.806394 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 02:41:42.806465 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:41:42.808151 22812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72: No bootstrap required, opened a new log
I20260620 02:41:42.819172 22812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55df05240e1044b195e076278d607e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35335 } }
I20260620 02:41:42.819654 22812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:41:42.819731 22812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55df05240e1044b195e076278d607e72, State: Initialized, Role: FOLLOWER
I20260620 02:41:42.820434 22812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55df05240e1044b195e076278d607e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35335 } }
I20260620 02:41:42.820722 22812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:41:42.820827 22812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:41:42.820952 22812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:42.823364 22812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55df05240e1044b195e076278d607e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35335 } }
I20260620 02:41:42.824051 22812 leader_election.cc:302] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55df05240e1044b195e076278d607e72; no voters:
I20260620 02:41:42.825006 22812 leader_election.cc:288] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:41:42.825212 22822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:42.826359 22822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [term 1 LEADER]: Becoming Leader. State: Replica: 55df05240e1044b195e076278d607e72, State: Running, Role: LEADER
I20260620 02:41:42.827204 22822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55df05240e1044b195e076278d607e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35335 } }
I20260620 02:41:42.828261 22812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:42.831338 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55df05240e1044b195e076278d607e72. Latest consensus state: current_term: 1 leader_uuid: "55df05240e1044b195e076278d607e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55df05240e1044b195e076278d607e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35335 } } }
I20260620 02:41:42.832132 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:42.831343 22823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55df05240e1044b195e076278d607e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55df05240e1044b195e076278d607e72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35335 } } }
I20260620 02:41:42.832787 22823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:42.834792 22829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:42.841449 22829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:42.852622 22829 catalog_manager.cc:1347] Generated new cluster ID: 8c81828a70ee40d9ac839607bb2f8a12
I20260620 02:41:42.852774 22829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:42.878365 22829 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:42.879194 22829 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:42.891247 22829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55df05240e1044b195e076278d607e72: Generated new TSK 0
I20260620 02:41:42.892798 22829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:41:42.914191 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22745
W20260620 02:41:42.916100 22807 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49899
W20260620 02:41:42.917646 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22745: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:41:42.917897 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 22745
2026-06-20T02:41:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2104 ms)
[----------] 18 tests from AuthTokenIssuingTest (40618 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:43 dist-test-slave-rpg6 krb5kdc[22843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:43 dist-test-slave-rpg6 krb5kdc[22843](info): set up 2 sockets
Jun 20 02:41:43 dist-test-slave-rpg6 krb5kdc[22843](info): commencing operation
krb5kdc: starting...
W20260620 02:41:44.073168 388 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:44 dist-test-slave-rpg6 krb5kdc[22843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:41:44.369524 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43207
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43207,127.0.97.61:38775,127.0.97.60:41387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:44.737802 22856 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:41:44.738291 22856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:44.738435 22856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:44.738574 22856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:44.746942 22856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:44.747119 22856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:44.747236 22856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:44.747344 22856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:44.747452 22856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:44.758980 22856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43207,127.0.97.61:38775,127.0.97.60:41387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:44.760524 22856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:44.761994 22856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:44.776147 22861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:44.776041 22862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:44.776173 22864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:44.777043 22856 server_base.cc:1034] running on GCE node
I20260620 02:41:44.777935 22856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:44.779645 22856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:44.780933 22856 hybrid_clock.cc:648] HybridClock initialized: now 1781923304780869 us; error 42 us; skew 500 ppm
I20260620 02:41:44.781327 22856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:44 dist-test-slave-rpg6 krb5kdc[22843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:44.791960 22856 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:41:44.794032 22856 webserver.cc:460] Webserver started at http://127.0.97.62:46363/ using document root <none> and password file <none>
I20260620 02:41:44.794844 22856 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:44.794991 22856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:44.795344 22856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:44.798780 22856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "e436693865524c8bb95c5e613a9bbb68"
format_stamp: "Formatted at 2026-06-20 02:41:44 on dist-test-slave-rpg6"
server_key: "531f8ee3772f0e288fc7d207e26a1870"
server_key_iv: "32b13891c743e2a7241593f43dc9bc16"
server_key_version: "encryptionkey@0"
I20260620 02:41:44.799765 22856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "e436693865524c8bb95c5e613a9bbb68"
format_stamp: "Formatted at 2026-06-20 02:41:44 on dist-test-slave-rpg6"
server_key: "531f8ee3772f0e288fc7d207e26a1870"
server_key_iv: "32b13891c743e2a7241593f43dc9bc16"
server_key_version: "encryptionkey@0"
I20260620 02:41:44.806664 22856 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:41:44.810858 22871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:44.812629 22856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:41:44.812853 22856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "e436693865524c8bb95c5e613a9bbb68"
format_stamp: "Formatted at 2026-06-20 02:41:44 on dist-test-slave-rpg6"
server_key: "531f8ee3772f0e288fc7d207e26a1870"
server_key_iv: "32b13891c743e2a7241593f43dc9bc16"
server_key_version: "encryptionkey@0"
I20260620 02:41:44.813124 22856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:44.841961 22856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:44.847124 22856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:44.847455 22856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:44.869843 22856 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43207
I20260620 02:41:44.869867 22922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43207 every 8 connection(s)
I20260620 02:41:44.871425 22856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:44.875780 22923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:44.881871 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 22856
I20260620 02:41:44.882157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:44.883131 388 external_mini_cluster.cc:1428] Setting key 7935a4c95d052402a5edf82dc840325a
I20260620 02:41:44.883507 22923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:41:44 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260620 02:41:44.911159 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:44.888424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43207 (local address 127.0.0.1:58006)
0620 02:41:44.889456 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:44.889504 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:44.892577 (+ 3073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:44.893635 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:44.893642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:44.894447 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:44.894815 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:44.894835 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:44.896614 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:44.896619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:44.897807 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:44.897821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:44.898852 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:44.898859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:44.899025 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:44.899950 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:44.900008 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:44.902450 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:44.906365 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:44.906379 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:44.906393 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:44.906842 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:44.907306 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:44.907311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:44.907314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:44.907473 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:44.908405 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:44.908414 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:44.908626 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 02:41:44.908898 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 02:41:44.909657 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":219,"thread_start_us":446,"threads_started":1}
I20260620 02:41:44.911139 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:44.888424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.0.97.62:43207)
0620 02:41:44.890684 (+ 2260us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:44.890694 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:44.890770 (+ 76us) server_negotiation.cc:407] Connection header received
0620 02:41:44.893060 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:44.893067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:44.893371 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:44.893610 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:44.895060 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:44.896437 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:44.897988 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:44.898587 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:44.898812 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:44.902736 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:44.902746 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:44.902761 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:44.902796 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:44.906130 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:44.906996 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:44.907004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:44.907009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:44.907166 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:44.907596 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:44.907600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:44.907605 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:44.908256 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:44.908476 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:44.909031 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 02:41:44.909551 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":1993,"thread_start_us":2105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 02:41:44.915144 22923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:41:44.916728 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:44.916155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39404)
0620 02:41:44.916356 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:44.916492 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 02:41:44.917120 22873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
I20260620 02:41:44.919353 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:44.918698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39416)
0620 02:41:44.918950 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:44.919111 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 02:41:44.919741 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38775: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 02:41:44.951488 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38775
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43207,127.0.97.61:38775,127.0.97.60:41387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:41:44.969215 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } attempt: 1
I20260620 02:41:44.970609 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:44.970170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39432)
0620 02:41:44.970370 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:44.970474 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 02:41:44.971712 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:44.971343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39442)
0620 02:41:44.971507 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:44.971575 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:41:44.971926 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38775: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
I20260620 02:41:45.040522 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } attempt: 2
I20260620 02:41:45.041985 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.041358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39452)
0620 02:41:45.041699 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.041825 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 02:41:45.043231 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.042760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39466)
0620 02:41:45.042969 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.043071 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260620 02:41:45.043419 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38775: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
I20260620 02:41:45.134838 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } attempt: 3
I20260620 02:41:45.136579 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.136047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39468)
0620 02:41:45.136327 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.136423 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 02:41:45.138309 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.137868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39482)
0620 02:41:45.138049 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.138130 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 02:41:45.138525 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38775: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
I20260620 02:41:45.281831 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } attempt: 4
I20260620 02:41:45.283402 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.282714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39496)
0620 02:41:45.283129 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.283258 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260620 02:41:45.284618 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.284206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39498)
0620 02:41:45.284376 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.284476 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 02:41:45.284927 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38775: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38775: connect: Connection refused (error 111)
W20260620 02:41:45.326195 22930 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:41:45.326738 22930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:45.327037 22930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:45.327208 22930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:45.335227 22930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:45.335420 22930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:45.335538 22930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:45.335628 22930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:45.335747 22930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:45.348052 22930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43207,127.0.97.61:38775,127.0.97.60:41387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:45.349637 22930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:45.351179 22930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:45.364810 22936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:45.364820 22938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:45.364845 22935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:45.370086 22930 server_base.cc:1034] running on GCE node
I20260620 02:41:45.371089 22930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:45.372787 22930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:45.374114 22930 hybrid_clock.cc:648] HybridClock initialized: now 1781923305374073 us; error 59 us; skew 500 ppm
I20260620 02:41:45.374534 22930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:45 dist-test-slave-rpg6 krb5kdc[22843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:45.378253 22930 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 02:41:45.380225 22930 webserver.cc:460] Webserver started at http://127.0.97.61:45517/ using document root <none> and password file <none>
I20260620 02:41:45.381034 22930 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:45.381170 22930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:45.381510 22930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:45.384953 22930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/data/instance:
uuid: "a90f6a678e8e4bcdb2a43b135723e912"
format_stamp: "Formatted at 2026-06-20 02:41:45 on dist-test-slave-rpg6"
server_key: "51453dee7c064fb0ec3922d2d4b88c8b"
server_key_iv: "8dcfdcf8b36497b614e6ac094559f883"
server_key_version: "encryptionkey@0"
I20260620 02:41:45.385962 22930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/wal/instance:
uuid: "a90f6a678e8e4bcdb2a43b135723e912"
format_stamp: "Formatted at 2026-06-20 02:41:45 on dist-test-slave-rpg6"
server_key: "51453dee7c064fb0ec3922d2d4b88c8b"
server_key_iv: "8dcfdcf8b36497b614e6ac094559f883"
server_key_version: "encryptionkey@0"
I20260620 02:41:45.392366 22930 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:41:45.396427 22945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:45.398267 22930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:41:45.398515 22930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/wal
uuid: "a90f6a678e8e4bcdb2a43b135723e912"
format_stamp: "Formatted at 2026-06-20 02:41:45 on dist-test-slave-rpg6"
server_key: "51453dee7c064fb0ec3922d2d4b88c8b"
server_key_iv: "8dcfdcf8b36497b614e6ac094559f883"
server_key_version: "encryptionkey@0"
I20260620 02:41:45.398785 22930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:45.424937 22930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:45.430876 22930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:45.431352 22930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:45.453819 22930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38775
I20260620 02:41:45.453842 22996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38775 every 8 connection(s)
I20260620 02:41:45.455376 22930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:41:45.459524 22997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:45.462486 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 22930
I20260620 02:41:45.462688 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-1/wal/instance
I20260620 02:41:45.463176 388 external_mini_cluster.cc:1428] Setting key 7b6f17c4562c659ac61308f8fe92a6a1
I20260620 02:41:45.469508 22997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:41:45 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 02:41:45.497148 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.473492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.0.97.62:43207)
0620 02:41:45.474168 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:45.474175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:45.475696 (+ 1521us) server_negotiation.cc:407] Connection header received
0620 02:41:45.477695 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:45.477705 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:45.477813 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:45.477950 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:45.479764 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.480967 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.482935 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.483747 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.483997 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:45.488607 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:45.488618 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:45.488625 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:45.488697 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:45.492934 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:45.493867 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:45.493872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:45.493875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:45.494004 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:45.494522 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:45.494528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:45.494532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:45.494927 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:45.495186 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:45.496502 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0620 02:41:45.496754 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":383,"threads_started":1}
I20260620 02:41:45.498219 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.474326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43207 (local address 127.0.0.1:58010)
0620 02:41:45.475302 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.475371 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:45.477339 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:45.478173 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:45.478183 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:45.478816 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:45.479447 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:45.479467 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.481210 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.481216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:45.482728 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:45.482747 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.484163 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.484171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:45.484429 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:45.485330 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:45.485382 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:45.488267 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:45.493126 (+ 4859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:45.493135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:45.493152 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:45.493725 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:45.494123 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:45.494126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:45.494129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:45.494348 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:45.495526 (+ 1178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:45.495553 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:45.496101 (+ 548us) client_negotiation.cc:769] Sending connection context
0620 02:41:45.496415 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 02:41:45.496971 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":375,"threads_started":1}
I20260620 02:41:45.500345 22997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:41:45 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:41:45.519208 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.501406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39510)
0620 02:41:45.501712 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.501737 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:45.501902 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:45.502996 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:45.503000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:45.503584 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:45.503883 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:45.503897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.506124 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.506132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:45.507478 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:45.507494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.508590 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.508597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:45.508793 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:45.509511 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:45.509529 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:45.512195 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:45.516181 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:45.516186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:45.516190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:45.516672 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:45.517130 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:45.517136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:45.517139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:45.517285 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:45.518102 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:45.518108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:45.518282 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 02:41:45.518429 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 02:41:45.518773 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 02:41:45.519403 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.501425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.0.97.61:38775)
0620 02:41:45.502373 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:45.502384 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:45.502420 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:41:45.502485 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:45.502497 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:45.502805 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:45.502972 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:45.504076 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.505904 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.507660 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.508422 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.508586 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:45.512451 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:45.512467 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:45.512484 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:45.512541 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:45.515998 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:45.516813 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:45.516823 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:45.516826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:45.517006 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:45.517389 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:45.517397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:45.517401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:45.517912 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:45.518160 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:45.518584 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 02:41:45.518900 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":716,"thread_start_us":515,"threads_started":1}
I20260620 02:41:45.522179 22997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:41:45.523679 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.523101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47014)
0620 02:41:45.523380 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.523504 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 02:41:45.524163 22949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
I20260620 02:41:45.526535 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.525865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47028)
0620 02:41:45.526237 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.526371 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260620 02:41:45.526896 22997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 02:41:45.536960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41387
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43207,127.0.97.61:38775,127.0.97.60:41387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:41:45.576256 22997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } attempt: 1
I20260620 02:41:45.577978 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.577077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47040)
0620 02:41:45.577438 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.577737 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260620 02:41:45.579156 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.578749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47044)
0620 02:41:45.578942 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.579035 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 02:41:45.579447 22997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
I20260620 02:41:45.584209 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } attempt: 5
Jun 20 02:41:45 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:41:45.601915 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.584996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39512)
0620 02:41:45.585249 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.585275 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:45.585428 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:45.585968 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:45.585976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:45.586548 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:45.586927 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:45.586942 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.588204 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.588210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:45.589625 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:45.589646 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.590525 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.590533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:45.590751 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:45.591480 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:45.591504 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:45.594011 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:45.598977 (+ 4966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:45.598982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:45.598995 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:45.599482 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:45.600050 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:45.600054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:45.600056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:45.600207 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:45.600990 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:45.600994 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:45.601197 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 02:41:45.601331 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 02:41:45.601655 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 02:41:45.602074 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.585110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.0.97.61:38775)
0620 02:41:45.585298 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:45.585302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:45.585329 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:41:45.585695 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:45.585702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:45.585779 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:45.585871 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:45.587144 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.588040 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.589819 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:45.590335 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:45.590441 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:45.594432 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:45.594441 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:45.594445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:45.594492 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:45.598741 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:45.599646 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:45.599654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:45.599658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:45.599897 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:45.600336 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:45.600343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:45.600348 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:45.600815 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:45.600968 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:45.601476 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 02:41:45.601747 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:41:45.603235 22923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:41:45.604775 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.604267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47052)
0620 02:41:45.604439 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.604523 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 02:41:45.606035 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.605613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47066)
0620 02:41:45.605776 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.605882 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 02:41:45.606319 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
I20260620 02:41:45.647730 22997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } attempt: 2
I20260620 02:41:45.649044 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.648567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47070)
0620 02:41:45.648786 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.648876 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 02:41:45.650314 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.649920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47082)
0620 02:41:45.650104 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.650189 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 02:41:45.650691 22997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
I20260620 02:41:45.657593 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } attempt: 1
I20260620 02:41:45.658991 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.658435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47098)
0620 02:41:45.658661 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.658778 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 02:41:45.660158 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.659730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47110)
0620 02:41:45.659918 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.660006 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:41:45.660521 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
I20260620 02:41:45.728842 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } attempt: 2
I20260620 02:41:45.730682 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.729883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47118)
0620 02:41:45.730181 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.730498 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 02:41:45.732254 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.731654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47126)
0620 02:41:45.731922 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.732116 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 02:41:45.732497 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
I20260620 02:41:45.742045 22997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } attempt: 3
I20260620 02:41:45.743783 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.743186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47138)
0620 02:41:45.743497 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.743598 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260620 02:41:45.745177 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.744666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47152)
0620 02:41:45.744914 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.745011 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260620 02:41:45.745393 22997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
I20260620 02:41:45.838762 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } attempt: 3
I20260620 02:41:45.840070 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.839456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47168)
0620 02:41:45.839747 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.839888 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:41:45.841228 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.840775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47180)
0620 02:41:45.841039 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.841120 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 02:41:45.841420 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
I20260620 02:41:45.888708 22997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } attempt: 4
I20260620 02:41:45.890332 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.889571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47184)
0620 02:41:45.889895 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.889994 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 02:41:45.891752 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:45.891362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47196)
0620 02:41:45.891537 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:45.891627 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 02:41:45.892014 22997 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
W20260620 02:41:45.922742 23005 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:41:45.923184 23005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:45.923322 23005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:45.923460 23005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:45.931097 23005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:45.931279 23005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:45.931454 23005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:45.931675 23005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:45.931811 23005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:45.942983 23005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43207,127.0.97.61:38775,127.0.97.60:41387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:45.944609 23005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:45.946077 23005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:45.959784 23010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:45.960186 23011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:45.959874 23013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:45.961061 23005 server_base.cc:1034] running on GCE node
I20260620 02:41:45.961993 23005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:45.963618 23005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:45.964911 23005 hybrid_clock.cc:648] HybridClock initialized: now 1781923305964843 us; error 81 us; skew 500 ppm
I20260620 02:41:45.965340 23005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:45 dist-test-slave-rpg6 krb5kdc[22843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:45.969492 23005 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 02:41:45.971616 23005 webserver.cc:460] Webserver started at http://127.0.97.60:34019/ using document root <none> and password file <none>
I20260620 02:41:45.972558 23005 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:45.972702 23005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:45.973060 23005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:45.976468 23005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/data/instance:
uuid: "332728c6321c458f9f29379c1ea3f081"
format_stamp: "Formatted at 2026-06-20 02:41:45 on dist-test-slave-rpg6"
server_key: "d95066deddf6da0ff51852cbb0688ed1"
server_key_iv: "6589bf53268513e351085ce1d8d42b7d"
server_key_version: "encryptionkey@0"
I20260620 02:41:45.977414 23005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/wal/instance:
uuid: "332728c6321c458f9f29379c1ea3f081"
format_stamp: "Formatted at 2026-06-20 02:41:45 on dist-test-slave-rpg6"
server_key: "d95066deddf6da0ff51852cbb0688ed1"
server_key_iv: "6589bf53268513e351085ce1d8d42b7d"
server_key_version: "encryptionkey@0"
I20260620 02:41:45.984961 23005 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260620 02:41:45.989768 23020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:45.991637 23005 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:41:45.991946 23005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/wal
uuid: "332728c6321c458f9f29379c1ea3f081"
format_stamp: "Formatted at 2026-06-20 02:41:45 on dist-test-slave-rpg6"
server_key: "d95066deddf6da0ff51852cbb0688ed1"
server_key_iv: "6589bf53268513e351085ce1d8d42b7d"
server_key_version: "encryptionkey@0"
I20260620 02:41:45.992270 23005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:46.008419 23005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:46.013330 23005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:46.013666 23005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:46.018616 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } attempt: 4
I20260620 02:41:46.019905 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.019332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47200)
0620 02:41:46.019558 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:46.019681 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:41:46.021185 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.020632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47214)
0620 02:41:46.020888 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:46.020990 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 02:41:46.021468 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41387: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41387: connect: Connection refused (error 111)
I20260620 02:41:46.035276 23005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41387
I20260620 02:41:46.035298 23071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41387 every 8 connection(s)
I20260620 02:41:46.036851 23005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:41:46.037472 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 23005
I20260620 02:41:46.037662 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781922915737669-388-0/minicluster-data/master-2/wal/instance
I20260620 02:41:46.038257 388 external_mini_cluster.cc:1428] Setting key f37a4cf4f7dcf025df3278e19a42a4fb
I20260620 02:41:46.041811 23072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:46.049292 23072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:41:46 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:41:46 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:41:46.063243 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.040428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.0.97.60:41387)
0620 02:41:46.041281 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.041294 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.041337 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:41:46.043155 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.043180 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.043638 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.044162 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.045850 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.047325 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.048818 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.049559 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.049794 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.054700 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:46.054708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:46.054720 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:46.054762 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:46.058227 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.059045 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.059076 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.059088 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.059346 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.059805 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.059811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.059814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.060466 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:46.060849 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:46.061312 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 02:41:46.061888 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":391,"threads_started":1}
I20260620 02:41:46.070842 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.052241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.0.97.62:43207)
0620 02:41:46.052927 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.052933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.053548 (+ 615us) server_negotiation.cc:407] Connection header received
0620 02:41:46.053714 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.053721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.053806 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.053944 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.055262 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.056158 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.057710 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.058354 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.058548 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.062644 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:46.062652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:46.062656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:46.062701 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:46.066303 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.067460 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.067469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.067473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.067747 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.068493 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.068500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.068508 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.068901 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:46.069121 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:46.069833 (+ 712us) server_negotiation.cc:299] Negotiation successful
0620 02:41:46.070238 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":338,"threads_started":1}
I20260620 02:41:46.070827 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.052241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43207 (local address 127.0.0.1:58018)
0620 02:41:46.053310 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:46.053361 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:46.053566 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:46.054043 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:46.054048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:46.054741 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:46.055065 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:46.055075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.056343 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.056347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:46.057506 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:46.057521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.058593 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.058602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:46.058810 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.059772 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:46.059791 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:46.062382 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:46.066522 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:46.066535 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:46.066542 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:46.067199 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:46.068071 (+ 872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:46.068078 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:46.068082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:46.068308 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:46.069136 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:46.069150 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:46.069414 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 02:41:46.069695 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 02:41:46.070342 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":628,"threads_started":1}
I20260620 02:41:46.072930 23072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:41:46 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 02:41:46 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:41:46.086127 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.067771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.0.97.62:43207)
0620 02:41:46.068564 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.068569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.068598 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:41:46.068668 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.068672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.068738 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.068887 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.070408 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.072386 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.074246 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.074684 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.074849 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.079611 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:46.079620 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:46.079626 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:46.079673 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:46.083067 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.083910 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.083915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.083919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.084068 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.084486 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.084491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.084496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.084818 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:46.085109 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:46.085475 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 02:41:46.085720 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":391,"threads_started":1}
I20260620 02:41:46.090570 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.073897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38775 (local address 127.0.0.1:39518)
0620 02:41:46.074081 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:46.074104 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:46.074229 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:46.074580 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:46.074585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:46.075006 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:46.075184 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:46.075193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.076298 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.076303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:46.077206 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:46.077218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.077973 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.077981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:46.078191 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.078856 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:46.078881 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:46.081821 (+ 2940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:46.087305 (+ 5484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:46.087310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:46.087319 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:46.087754 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:46.088481 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:46.088486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:46.088490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:46.088693 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:46.089577 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:46.089583 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:46.089837 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 02:41:46.090028 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 02:41:46.090301 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 02:41:46.091678 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.074088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.0.97.61:38775)
0620 02:41:46.074295 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.074301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.074322 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:41:46.074383 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.074387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.074446 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.074553 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.075370 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.076140 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.077367 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.077794 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.077935 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.082224 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:46.082233 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:46.082238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:46.082301 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:46.087080 (+ 4779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.088052 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.088059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.088062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.088324 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.088857 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.088862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.088865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.089330 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:46.089472 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:46.090607 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0620 02:41:46.091187 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:41:46.093708 23072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:41:46 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 02:41:46 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:41:46.113629 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.088793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39534 (local address 127.0.97.61:38775)
0620 02:41:46.089420 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.089426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.089448 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:41:46.089495 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.089503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.089578 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.089684 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.092387 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.094090 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.096601 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.097487 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.097633 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.104685 (+ 7052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:46.104692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:46.104696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:46.104748 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:46.109361 (+ 4613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.110339 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.110346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.110350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.110520 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.111385 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.111389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.111391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.111991 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:46.112296 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:46.112869 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 02:41:46.113159 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":280,"threads_started":1}
I20260620 02:41:46.114337 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.094692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47224)
0620 02:41:46.094997 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:46.095022 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:46.095217 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:46.095595 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:46.095600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:46.096209 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:46.096399 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:46.096414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.097890 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.097896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:46.099687 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:46.099737 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.101084 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.101095 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:46.101591 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.102997 (+ 1406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:46.103017 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:46.106400 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:46.110807 (+ 4407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:46.110815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:46.110820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:46.111677 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:46.112302 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:46.112309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:46.112325 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:46.112544 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:46.113366 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:46.113369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:46.113497 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 02:41:46.113611 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 02:41:46.113968 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260620 02:41:46.114630 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.094859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.0.97.60:41387)
0620 02:41:46.095018 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.095023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.095152 (+ 129us) server_negotiation.cc:407] Connection header received
0620 02:41:46.095362 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.095366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.095428 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.095506 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.096583 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.097742 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.100161 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.100657 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.100894 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.106734 (+ 5840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:46.106742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:46.106748 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:46.106809 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:46.110539 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.111923 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.111930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.111934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.112144 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.112816 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.112822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.112826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.113229 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:46.113397 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:46.113779 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 02:41:46.114222 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":64}
I20260620 02:41:46.121047 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:46.121227 23072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081: Bootstrap starting.
I20260620 02:41:46.126091 23072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:46.127403 23072 log.cc:824] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:46.125928 22898 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e436693865524c8bb95c5e613a9bbb68"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58028
I20260620 02:41:46.131572 23072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081: No bootstrap required, opened a new log
I20260620 02:41:46.142222 23072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } }
I20260620 02:41:46.142650 23072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332728c6321c458f9f29379c1ea3f081, State: Initialized, Role: FOLLOWER
I20260620 02:41:46.143260 23072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } }
I20260620 02:41:46.144917 23081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } } }
I20260620 02:41:46.145408 23081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:46.146427 23072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:41:46.157812 23092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:41:46.158052 23092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:41:46.155169 23005 master_runner.cc:186] Error getting master registration for 127.0.97.62:43207: OK, instance_id { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" instance_seqno: 1781923304863430 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:41:46.167485 23005 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:41:46.191275 22997 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } attempt: 5
Jun 20 02:41:46 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:41:46.208679 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.192353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.0.97.60:41387)
0620 02:41:46.192559 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.192565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.192599 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:41:46.192717 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.192724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.192815 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.192943 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.194088 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.195316 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.196976 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.197726 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.197870 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.201761 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:46.201769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:46.201774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:46.201830 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:46.205521 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.206379 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.206385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.206389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.206536 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.206978 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.206984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.206987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.207421 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:46.207642 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:46.208025 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 02:41:46.208261 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 02:41:46.208668 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.192181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47236)
0620 02:41:46.192405 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:46.192429 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:46.192570 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:46.192978 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:46.192985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:46.193540 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:46.193886 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:46.193898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.195481 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.195488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:46.196818 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:46.196835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.197910 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.197919 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:46.198123 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.198911 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:46.198929 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:46.201474 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:46.205735 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:46.205742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:46.205746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:46.206211 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:46.206686 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:46.206692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:46.206696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:46.206857 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:46.207561 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:46.207566 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:46.207742 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 02:41:46.207925 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 02:41:46.208262 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 02:41:46.213564 22997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912: Bootstrap starting.
I20260620 02:41:46.218832 22997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:46.220151 22997 log.cc:824] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:46.223408 22997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912: No bootstrap required, opened a new log
I20260620 02:41:46.231431 22997 raft_consensus.cc:348] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } }
I20260620 02:41:46.231812 22997 raft_consensus.cc:729] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90f6a678e8e4bcdb2a43b135723e912, State: Initialized, Role: FOLLOWER
I20260620 02:41:46.232417 22997 consensus_queue.cc:260] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } }
I20260620 02:41:46.233505 23095 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } } }
I20260620 02:41:46.233973 23095 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:46.234938 22997 sys_catalog.cc:564] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:46.243306 22930 master_runner.cc:186] Error getting master registration for 127.0.97.62:43207: OK, instance_id { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" instance_seqno: 1781923304863430 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:41:46.255779 22930 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 02:41:46.259490 23106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:41:46.259760 23106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 02:41:46.266191 23026 tablet.cc:2369] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:41:46.298734 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } attempt: 5
Jun 20 02:41:46 dist-test-slave-rpg6 krb5kdc[22843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:41:46.317644 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.299581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41387 (local address 127.0.0.1:47248)
0620 02:41:46.299992 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:46.300014 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:46.300143 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:46.300731 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:46.300739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:46.301189 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:46.301389 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:46.301398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.302879 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.302885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:46.304457 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:46.304473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.305719 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.305726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:46.305942 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.306662 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:46.306683 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:46.309583 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:46.314462 (+ 4879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:46.314470 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:46.314474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:46.315034 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:46.315521 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:46.315527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:46.315531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:46.315715 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:46.316433 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:46.316438 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:46.316611 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 02:41:46.316770 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 02:41:46.317145 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274}
I20260620 02:41:46.317644 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.299606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.0.97.60:41387)
0620 02:41:46.300385 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.300393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.300413 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:41:46.300476 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.300482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.300584 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.300688 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.301560 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.302734 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.304696 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.305547 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.305693 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.309848 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:46.309853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:46.309862 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:46.309901 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:46.314271 (+ 4370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.315205 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.315212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.315216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.315409 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.315893 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.315899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.315903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.316303 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:46.316461 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:46.316894 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 02:41:46.317229 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644}
I20260620 02:41:46.323196 22923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68: Bootstrap starting.
I20260620 02:41:46.327996 22923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:46.329648 22923 log.cc:824] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:46.333981 22923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68: No bootstrap required, opened a new log
I20260620 02:41:46.343217 22923 raft_consensus.cc:348] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } }
I20260620 02:41:46.343691 22923 raft_consensus.cc:729] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e436693865524c8bb95c5e613a9bbb68, State: Initialized, Role: FOLLOWER
I20260620 02:41:46.344326 22923 consensus_queue.cc:260] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } }
I20260620 02:41:46.345775 23109 sys_catalog.cc:455] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } } }
I20260620 02:41:46.346611 23109 sys_catalog.cc:458] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:46.347209 22923 sys_catalog.cc:564] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:46.347288 22898 raft_consensus.cc:482] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 02:41:46.347496 22898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:41:46.351274 22877 tablet.cc:2369] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:41:46.352506 22898 raft_consensus.cc:504] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } }
I20260620 02:41:46.355996 22972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e436693865524c8bb95c5e613a9bbb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a90f6a678e8e4bcdb2a43b135723e912"
I20260620 02:41:46.356751 22972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:46.362926 22972 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e436693865524c8bb95c5e613a9bbb68 in term 1.
I20260620 02:41:46.364172 22873 leader_election.cc:302] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a90f6a678e8e4bcdb2a43b135723e912, e436693865524c8bb95c5e613a9bbb68; no voters:
I20260620 02:41:46.366511 23047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e436693865524c8bb95c5e613a9bbb68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "332728c6321c458f9f29379c1ea3f081"
I20260620 02:41:46.367277 23047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:46.372786 23047 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e436693865524c8bb95c5e613a9bbb68 in term 1.
I20260620 02:41:46.373900 22898 leader_election.cc:288] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [CANDIDATE]: Term 1 election: Requested vote from peers a90f6a678e8e4bcdb2a43b135723e912 (127.0.97.61:38775), 332728c6321c458f9f29379c1ea3f081 (127.0.97.60:41387)
I20260620 02:41:46.381680 23109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:46.396348 23109 raft_consensus.cc:686] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [term 1 LEADER]: Becoming Leader. State: Replica: e436693865524c8bb95c5e613a9bbb68, State: Running, Role: LEADER
I20260620 02:41:46.398972 23109 consensus_queue.cc:237] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } }
W20260620 02:41:46.434999 22951 tablet.cc:2369] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:41:46.440371 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.405373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39536 (local address 127.0.97.61:38775)
0620 02:41:46.405648 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.405656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.419212 (+ 13556us) server_negotiation.cc:407] Connection header received
0620 02:41:46.419601 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.419611 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.419757 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.419904 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.422803 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.424051 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.425983 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.426721 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.426816 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.430854 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:46.430865 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:46.430871 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:46.430925 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:46.436607 (+ 5682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.437575 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.437581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.437585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.437758 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.438287 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.438294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.438297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.438733 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:46.438987 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:46.439581 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 02:41:46.440014 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:41:46.444080 23116 sys_catalog.cc:455] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e436693865524c8bb95c5e613a9bbb68. Latest consensus state: current_term: 1 leader_uuid: "e436693865524c8bb95c5e613a9bbb68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } } }
I20260620 02:41:46.445503 23116 sys_catalog.cc:458] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:46.469154 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.406822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.0.97.62:43207)
0620 02:41:46.407059 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.407067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.423749 (+ 16682us) server_negotiation.cc:407] Connection header received
0620 02:41:46.426127 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.426134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.426218 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.426317 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.434942 (+ 8625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.436283 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.447460 (+ 11177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.448629 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.448777 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.456040 (+ 7263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:46.456047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:46.456051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:46.456097 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:46.459918 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.465857 (+ 5939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.465864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.465867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.466060 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.466551 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.466556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.466559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.466962 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:46.467224 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:46.468498 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0620 02:41:46.468779 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 02:41:46.473845 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:46.409865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.0.97.60:41387)
0620 02:41:46.410126 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.410135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.424280 (+ 14145us) server_negotiation.cc:407] Connection header received
0620 02:41:46.426582 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.426590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.426686 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.426777 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.431166 (+ 4389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.432342 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.444813 (+ 12471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:46.445831 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:46.445962 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:46.460917 (+ 14955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:46.460924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:46.460929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:46.460973 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:46.464767 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.467756 (+ 2989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.467762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.467766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.467962 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:46.471737 (+ 3775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:46.471743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:46.471749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:46.472226 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:46.472496 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:46.473186 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 02:41:46.473483 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 02:41:46.474797 23129 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:41:46.475004 23129 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:41:46.475133 23130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:41:46.477429 23130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:46.502795 23047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [term 1 FOLLOWER]: Refusing update from remote peer e436693865524c8bb95c5e613a9bbb68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:41:46.502921 22972 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [term 1 FOLLOWER]: Refusing update from remote peer e436693865524c8bb95c5e613a9bbb68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:41:46.504040 23109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:41:46.504875 23109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:41:46.537411 23095 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e436693865524c8bb95c5e613a9bbb68. Latest consensus state: current_term: 1 leader_uuid: "e436693865524c8bb95c5e613a9bbb68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } } }
I20260620 02:41:46.537775 23095 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:46.544755 23116 sys_catalog.cc:455] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e436693865524c8bb95c5e613a9bbb68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } } }
I20260620 02:41:46.545092 23116 sys_catalog.cc:458] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:46.548568 23081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e436693865524c8bb95c5e613a9bbb68. Latest consensus state: current_term: 1 leader_uuid: "e436693865524c8bb95c5e613a9bbb68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } } }
I20260620 02:41:46.548934 23081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:46.552129 23130 catalog_manager.cc:1347] Generated new cluster ID: 8d60539b7ad34ac1bf80d7fc9baf5521
I20260620 02:41:46.552271 23130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:46.571741 23095 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e436693865524c8bb95c5e613a9bbb68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } } }
I20260620 02:41:46.572012 23095 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:46.584280 23109 sys_catalog.cc:455] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e436693865524c8bb95c5e613a9bbb68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } } }
I20260620 02:41:46.584750 23109 sys_catalog.cc:458] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:46.587392 23135 mvcc.cc:205] Tried to move back new op lower bound from 7298757863412383744 to 7298757863067447296. Current Snapshot: MvccSnapshot[applied={T|T < 7298757863412383744}]
I20260620 02:41:46.593366 23081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e436693865524c8bb95c5e613a9bbb68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e436693865524c8bb95c5e613a9bbb68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43207 } } peers { permanent_uuid: "a90f6a678e8e4bcdb2a43b135723e912" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38775 } } peers { permanent_uuid: "332728c6321c458f9f29379c1ea3f081" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41387 } } }
I20260620 02:41:46.593699 23081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:46.608086 23130 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:46.610100 23130 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:46.635859 23130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e436693865524c8bb95c5e613a9bbb68: Generated new TSK 0
I20260620 02:41:46.637151 23130 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:41:46.781961 23079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:46.761990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.0.97.61:38775)
0620 02:41:46.762187 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.762197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.779644 (+ 17447us) server_negotiation.cc:407] Connection header received
0620 02:41:46.780259 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.780266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.780352 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.780422 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.781706 (+ 1284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39538: BlockingRecv error: recv got EOF from 127.0.0.1:39538 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 02:41:46.784293 23078 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:46.759721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.0.97.62:43207)
0620 02:41:46.759984 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.759991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.779189 (+ 19198us) server_negotiation.cc:407] Connection header received
0620 02:41:46.782547 (+ 3358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.782554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.782678 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.782753 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.783942 (+ 1189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58048: BlockingRecv error: recv got EOF from 127.0.0.1:58048 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260620 02:41:46.787429 23075 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:46.775248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.0.97.60:41387)
0620 02:41:46.775490 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.775498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.785451 (+ 9953us) server_negotiation.cc:407] Connection header received
0620 02:41:46.785847 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.785855 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.785951 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.786029 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.787170 (+ 1141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47264: BlockingRecv error: recv got EOF from 127.0.0.1:47264 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 02:41:46.830631 23079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:46.819013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.0.97.61:38775)
0620 02:41:46.819212 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.819219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.828431 (+ 9212us) server_negotiation.cc:407] Connection header received
0620 02:41:46.828999 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.829006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.829176 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.829263 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.830347 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39552: BlockingRecv error: recv got EOF from 127.0.0.1:39552 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 02:41:46.865128 23079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:46.862758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.0.97.61:38775)
0620 02:41:46.863058 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.863068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.863646 (+ 578us) server_negotiation.cc:407] Connection header received
0620 02:41:46.863911 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.863917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.864006 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.864131 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.864891 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39566: BlockingRecv error: recv got EOF from 127.0.0.1:39566 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 02:41:46.921213 23079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:46.911002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.0.97.61:38775)
0620 02:41:46.911250 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.911259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.919551 (+ 8292us) server_negotiation.cc:407] Connection header received
0620 02:41:46.919792 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.919799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.919975 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.920097 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.920931 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39568: BlockingRecv error: recv got EOF from 127.0.0.1:39568 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 02:41:46.931263 23079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:46.928990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.0.97.61:38775)
0620 02:41:46.929269 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.929280 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.929667 (+ 387us) server_negotiation.cc:407] Connection header received
0620 02:41:46.929918 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.929924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.930021 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.930147 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.930965 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39578: BlockingRecv error: recv got EOF from 127.0.0.1:39578 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260620 02:41:46.945221 23079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:46.942958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.0.97.61:38775)
0620 02:41:46.943203 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.943209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.943736 (+ 527us) server_negotiation.cc:407] Connection header received
0620 02:41:46.944007 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.944012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.944100 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.944221 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.944965 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39592: BlockingRecv error: recv got EOF from 127.0.0.1:39592 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 02:41:46.967705 23079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:46.965265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.0.97.61:38775)
0620 02:41:46.965718 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:46.965726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:46.965753 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:41:46.965840 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:46.965846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:46.965945 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:46.966083 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:46.967355 (+ 1272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39604: BlockingRecv error: recv got EOF from 127.0.0.1:39604 (error 108)
Metrics: {"server-negotiator.queue_time_us":231}
W20260620 02:41:47.005648 23079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:47.003349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39612 (local address 127.0.97.61:38775)
0620 02:41:47.003640 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:47.003650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:47.004104 (+ 454us) server_negotiation.cc:407] Connection header received
0620 02:41:47.004328 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:47.004332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:47.004407 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:47.004522 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:47.005409 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39612: BlockingRecv error: recv got EOF from 127.0.0.1:39612 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 02:41:47.077208 23079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:47.074097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.0.97.61:38775)
0620 02:41:47.074376 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:47.074384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:47.075095 (+ 711us) server_negotiation.cc:407] Connection header received
0620 02:41:47.075394 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:47.075399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:47.075507 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:47.075635 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:47.076960 (+ 1325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39618: BlockingRecv error: recv got EOF from 127.0.0.1:39618 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:41:47.160238 23092 catalog_manager.cc:1259] Loaded cluster ID: 8d60539b7ad34ac1bf80d7fc9baf5521
I20260620 02:41:47.160362 23092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081: loading cluster ID for follower catalog manager: success
I20260620 02:41:47.163226 23092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081: acquiring CA information for follower catalog manager: success
I20260620 02:41:47.165370 23092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 332728c6321c458f9f29379c1ea3f081: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 02:41:47.219290 23079 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:47.215940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.0.97.61:38775)
0620 02:41:47.216186 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:47.216194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:47.216597 (+ 403us) server_negotiation.cc:407] Connection header received
0620 02:41:47.216867 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:47.216875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:47.217100 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:47.217250 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:47.218749 (+ 1499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39630: BlockingRecv error: recv got EOF from 127.0.0.1:39630 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:41:47.262265 23106 catalog_manager.cc:1259] Loaded cluster ID: 8d60539b7ad34ac1bf80d7fc9baf5521
I20260620 02:41:47.262379 23106 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912: loading cluster ID for follower catalog manager: success
I20260620 02:41:47.265084 23106 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912: acquiring CA information for follower catalog manager: success
I20260620 02:41:47.267040 23106 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a90f6a678e8e4bcdb2a43b135723e912: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:41:47.490449 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:47.482336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.0.97.61:38775)
0620 02:41:47.482674 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:47.482680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:47.482844 (+ 164us) server_negotiation.cc:407] Connection header received
0620 02:41:47.483120 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:47.483125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:47.483309 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:47.483418 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:41:47.483962 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:47.485003 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:47.486974 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:47.487427 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:47.487568 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:47.488494 (+ 926us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:41:47.489170 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:41:47.489294 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:47.489806 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 02:41:47.490057 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260620 02:41:47.505832 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:47.498395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.0.97.60:41387)
0620 02:41:47.498911 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:47.498915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:47.499040 (+ 125us) server_negotiation.cc:407] Connection header received
0620 02:41:47.499485 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:47.499493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:47.499748 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:47.500099 (+ 351us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:41:47.500698 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:47.501423 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:47.502984 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:47.503479 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:47.503590 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:47.504475 (+ 885us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:41:47.505147 (+ 672us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:41:47.505266 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:47.505478 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 02:41:47.505561 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":308,"threads_started":1}
I20260620 02:41:47.526607 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:47.516479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.0.97.60:41387)
0620 02:41:47.516741 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:47.516750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:47.520648 (+ 3898us) server_negotiation.cc:407] Connection header received
0620 02:41:47.520796 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:47.520805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:47.520925 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:47.521035 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:41:47.521562 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:47.522416 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:47.524458 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:47.525092 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:47.525242 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:47.525749 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:41:47.525869 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:41:47.525992 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:47.526151 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 02:41:47.526254 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260620 02:41:47.527209 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:47.516713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.0.97.62:43207)
0620 02:41:47.518890 (+ 2177us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:47.518897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:47.519235 (+ 338us) server_negotiation.cc:407] Connection header received
0620 02:41:47.519524 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:47.519532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:47.519781 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:47.520009 (+ 228us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:41:47.520526 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:47.521788 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:47.523180 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:47.523759 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:47.523918 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:47.525810 (+ 1892us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:41:47.526300 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:41:47.526414 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:47.526615 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 02:41:47.526714 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2012,"thread_start_us":1858,"threads_started":1}
I20260620 02:41:47.529492 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:47.516337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.0.97.61:38775)
0620 02:41:47.516565 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:47.516574 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:47.521913 (+ 5339us) server_negotiation.cc:407] Connection header received
0620 02:41:47.523457 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:47.523463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:47.523556 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:47.523667 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:41:47.524398 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:47.525463 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:47.527363 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:47.528029 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:47.528162 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:47.528757 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:41:47.528864 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:41:47.528975 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:47.529116 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 02:41:47.529208 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:41:47.533586 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22856
I20260620 02:41:47.535729 22918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:47.652601 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 22856
I20260620 02:41:47.674850 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 22930
I20260620 02:41:47.676549 22992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:47.795553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 22930
I20260620 02:41:47.817818 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 23005
I20260620 02:41:47.819160 23067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:47.933919 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 23005
2026-06-20T02:41:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5020 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:41:48 dist-test-slave-rpg6 krb5kdc[23219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:41:48 dist-test-slave-rpg6 krb5kdc[23219](info): set up 2 sockets
Jun 20 02:41:48 dist-test-slave-rpg6 krb5kdc[23219](info): commencing operation
krb5kdc: starting...
W20260620 02:41:49.128742 388 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/test-user.keytab.
Jun 20 02:41:49 dist-test-slave-rpg6 krb5kdc[23219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:41:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:41:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:41:49.437947 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37015
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37015,127.0.97.61:34909,127.0.97.60:34063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:41:49.827569 23232 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:41:49.828092 23232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:49.828225 23232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:49.828372 23232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:49.838598 23232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:49.839095 23232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:49.839288 23232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:49.839404 23232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:49.839648 23232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:49.853097 23232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37015,127.0.97.61:34909,127.0.97.60:34063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:49.854655 23232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:49.856531 23232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:49.871351 23238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:49.872284 23237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:49.872510 23240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:49.873580 23232 server_base.cc:1034] running on GCE node
I20260620 02:41:49.874557 23232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:49.876269 23232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:49.877555 23232 hybrid_clock.cc:648] HybridClock initialized: now 1781923309877488 us; error 50 us; skew 500 ppm
I20260620 02:41:49.878033 23232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:49 dist-test-slave-rpg6 krb5kdc[23219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:49.887457 23232 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:41:49.889843 23232 webserver.cc:460] Webserver started at http://127.0.97.62:43703/ using document root <none> and password file <none>
I20260620 02:41:49.890787 23232 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:49.890949 23232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:49.891310 23232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:49.896127 23232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/data/instance:
uuid: "623926bf87cc43db8ab34fd2f01f4343"
format_stamp: "Formatted at 2026-06-20 02:41:49 on dist-test-slave-rpg6"
server_key: "21b43fffb9282090f162c54a6cdd99f0"
server_key_iv: "6be622e20aa12c3cd04b4a2a0c5610c0"
server_key_version: "encryptionkey@0"
I20260620 02:41:49.897555 23232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/wal/instance:
uuid: "623926bf87cc43db8ab34fd2f01f4343"
format_stamp: "Formatted at 2026-06-20 02:41:49 on dist-test-slave-rpg6"
server_key: "21b43fffb9282090f162c54a6cdd99f0"
server_key_iv: "6be622e20aa12c3cd04b4a2a0c5610c0"
server_key_version: "encryptionkey@0"
I20260620 02:41:49.906733 23232 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260620 02:41:49.911469 23247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:49.913738 23232 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:41:49.914023 23232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/wal
uuid: "623926bf87cc43db8ab34fd2f01f4343"
format_stamp: "Formatted at 2026-06-20 02:41:49 on dist-test-slave-rpg6"
server_key: "21b43fffb9282090f162c54a6cdd99f0"
server_key_iv: "6be622e20aa12c3cd04b4a2a0c5610c0"
server_key_version: "encryptionkey@0"
I20260620 02:41:49.914317 23232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:49.945751 23232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:49.950647 23232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:49.950979 23232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:49.974416 23232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37015
I20260620 02:41:49.974808 23298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37015 every 8 connection(s)
I20260620 02:41:49.976006 23232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:41:49.982230 23299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:49.983943 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 23232
I20260620 02:41:49.984176 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-0/wal/instance
I20260620 02:41:49.984817 388 external_mini_cluster.cc:1428] Setting key 0b9e15d593020abadb48ef6046f7b3da
I20260620 02:41:49.993635 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:41:50 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 02:41:50.022049 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:49.998193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.0.97.62:37015)
0620 02:41:50.000150 (+ 1957us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:50.000165 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:50.000224 (+ 59us) server_negotiation.cc:407] Connection header received
0620 02:41:50.001997 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:50.002004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:50.002295 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:50.002517 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:50.004349 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.005872 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.007435 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.008114 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.008345 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:50.012405 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:50.012414 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:50.012428 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:50.012487 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:50.016566 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:50.017407 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:50.017423 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:50.017426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:50.017589 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:50.018039 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:50.018046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:50.018053 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:50.018780 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:50.019013 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:50.019623 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 02:41:50.020087 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1720,"thread_start_us":1452,"threads_started":1}
I20260620 02:41:50.022038 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:49.998192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37015 (local address 127.0.0.1:43954)
0620 02:41:49.999190 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:49.999252 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:50.001525 (+ 2273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:50.002542 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:50.002549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:50.003558 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:50.004090 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:50.004118 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.006074 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.006083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:50.007259 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:50.007273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.008379 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.008385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:50.008560 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:50.009375 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:50.009400 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:50.012128 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:50.016749 (+ 4621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:50.016763 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:50.016777 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:50.017236 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:50.017723 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:50.017729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:50.017733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:50.017916 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:50.018928 (+ 1012us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:50.018936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:50.019202 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 02:41:50.019477 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 02:41:50.020217 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":175,"thread_start_us":453,"threads_started":1}
I20260620 02:41:50.026993 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:41:50.028499 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.027973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43080)
0620 02:41:50.028237 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.028356 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260620 02:41:50.028880 23248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
I20260620 02:41:50.031205 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.030704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43090)
0620 02:41:50.030918 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.031027 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 02:41:50.031483 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34909: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 02:41:50.061162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34909
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37015,127.0.97.61:34909,127.0.97.60:34063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:41:50.080946 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } attempt: 1
I20260620 02:41:50.082468 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.081849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43106)
0620 02:41:50.082126 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.082256 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 02:41:50.084512 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.083324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43116)
0620 02:41:50.083909 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.084090 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
W20260620 02:41:50.085013 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34909: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
I20260620 02:41:50.153451 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } attempt: 2
I20260620 02:41:50.155457 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.154528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43128)
0620 02:41:50.154981 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.155195 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260620 02:41:50.157119 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.156573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43132)
0620 02:41:50.156781 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.156902 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 02:41:50.157425 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34909: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
I20260620 02:41:50.248859 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } attempt: 3
I20260620 02:41:50.250713 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.249746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43144)
0620 02:41:50.250110 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.250258 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260620 02:41:50.252671 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.251803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43146)
0620 02:41:50.252258 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.252399 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283}
W20260620 02:41:50.253127 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34909: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
I20260620 02:41:50.396479 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } attempt: 4
I20260620 02:41:50.398000 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.397255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43154)
0620 02:41:50.397673 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.397797 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260620 02:41:50.399264 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.398831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43168)
0620 02:41:50.399014 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.399097 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 02:41:50.399679 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34909: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34909: connect: Connection refused (error 111)
W20260620 02:41:50.457086 23306 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:41:50.457491 23306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:50.457630 23306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:50.457777 23306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:50.466117 23306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:50.466226 23306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:50.466306 23306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:50.466414 23306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:50.466522 23306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:50.477389 23306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37015,127.0.97.61:34909,127.0.97.60:34063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:50.479526 23306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:50.481169 23306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:50.496229 23312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:50.496337 23311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:50.496337 23314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:50.506201 23306 server_base.cc:1034] running on GCE node
I20260620 02:41:50.507304 23306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:50.509079 23306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:50.510380 23306 hybrid_clock.cc:648] HybridClock initialized: now 1781923310510321 us; error 41 us; skew 500 ppm
I20260620 02:41:50.510864 23306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:50 dist-test-slave-rpg6 krb5kdc[23219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:50.514410 23306 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 02:41:50.516510 23306 webserver.cc:460] Webserver started at http://127.0.97.61:43511/ using document root <none> and password file <none>
I20260620 02:41:50.517347 23306 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:50.517450 23306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:50.517839 23306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:50.521507 23306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/data/instance:
uuid: "4ed2adf95ccf4d2bbb4853ebce43b506"
format_stamp: "Formatted at 2026-06-20 02:41:50 on dist-test-slave-rpg6"
server_key: "e79b4620b4d1c1a809aa9113f38ba2fe"
server_key_iv: "e56bb2933ea29d1b1045af273d1b229c"
server_key_version: "encryptionkey@0"
I20260620 02:41:50.522461 23306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/wal/instance:
uuid: "4ed2adf95ccf4d2bbb4853ebce43b506"
format_stamp: "Formatted at 2026-06-20 02:41:50 on dist-test-slave-rpg6"
server_key: "e79b4620b4d1c1a809aa9113f38ba2fe"
server_key_iv: "e56bb2933ea29d1b1045af273d1b229c"
server_key_version: "encryptionkey@0"
I20260620 02:41:50.529541 23306 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 02:41:50.533829 23321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:50.536221 23306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:41:50.536453 23306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/wal
uuid: "4ed2adf95ccf4d2bbb4853ebce43b506"
format_stamp: "Formatted at 2026-06-20 02:41:50 on dist-test-slave-rpg6"
server_key: "e79b4620b4d1c1a809aa9113f38ba2fe"
server_key_iv: "e56bb2933ea29d1b1045af273d1b229c"
server_key_version: "encryptionkey@0"
I20260620 02:41:50.536698 23306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:50.565019 23306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:50.570377 23306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:50.570716 23306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:50.593468 23306 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34909
I20260620 02:41:50.593504 23372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34909 every 8 connection(s)
I20260620 02:41:50.594969 23306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:41:50.599217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 23306
I20260620 02:41:50.599601 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-1/wal/instance
I20260620 02:41:50.600234 388 external_mini_cluster.cc:1428] Setting key cdb16c0a9efbeb822380bb39d9a188d4
I20260620 02:41:50.600853 23373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:50.608152 23373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:41:50 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 02:41:50.636911 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.611476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.0.97.62:37015)
0620 02:41:50.612060 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:50.612066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:50.613295 (+ 1229us) server_negotiation.cc:407] Connection header received
0620 02:41:50.616503 (+ 3208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:50.616509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:50.616603 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:50.616735 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:50.619743 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.620558 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.623337 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.623802 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.624006 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:50.629331 (+ 5325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:50.629338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:50.629341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:50.629398 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:50.633173 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:50.634118 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:50.634124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:50.634128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:50.634276 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:50.634755 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:50.634763 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:50.634766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:50.635074 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:50.635259 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:50.636236 (+ 977us) server_negotiation.cc:299] Negotiation successful
0620 02:41:50.636501 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":316,"threads_started":1}
I20260620 02:41:50.638036 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.612244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37015 (local address 127.0.0.1:43962)
0620 02:41:50.613095 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.613148 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:50.616046 (+ 2898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:50.617381 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:50.617398 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:50.618414 (+ 1016us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:50.619354 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:50.619388 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.620768 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.620777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:50.623110 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:50.623133 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.624158 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.624170 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:50.624551 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:50.625491 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:50.625533 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:50.628999 (+ 3466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:50.633429 (+ 4430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:50.633439 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:50.633456 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:50.633966 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:50.634428 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:50.634433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:50.634435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:50.634633 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:50.635223 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:50.635232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:50.635661 (+ 429us) client_negotiation.cc:769] Sending connection context
0620 02:41:50.636026 (+ 365us) client_negotiation.cc:240] Negotiation successful
0620 02:41:50.636752 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":335,"threads_started":1}
I20260620 02:41:50.640384 23373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:41:50 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:41:50.660290 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.641468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43170)
0620 02:41:50.641774 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.641795 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:50.641927 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:50.643086 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:50.643091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:50.643654 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:50.643912 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:50.643930 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.645528 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.645536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:50.647110 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:50.647130 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.648854 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.648863 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:50.649150 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:50.649879 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:50.649908 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:50.653004 (+ 3096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:50.657416 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:50.657424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:50.657430 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:50.658028 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:50.658504 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:50.658512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:50.658517 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:50.658696 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:50.659366 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:50.659371 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:50.659529 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 02:41:50.659652 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 02:41:50.659951 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 02:41:50.660969 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.641786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.0.97.61:34909)
0620 02:41:50.642619 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:50.642625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:50.642654 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:41:50.642703 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:50.642710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:50.642909 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:50.643034 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:50.644181 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.645324 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.647308 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.648566 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.648814 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:50.653337 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:50.653354 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:50.653370 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:50.653409 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:50.657227 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:50.658233 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:50.658241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:50.658243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:50.658379 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:50.658825 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:50.658829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:50.658831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:50.659221 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:50.659393 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:50.659916 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 02:41:50.660209 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":512,"threads_started":1}
I20260620 02:41:50.664039 23373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:41:50.665374 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.664846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38240)
0620 02:41:50.665036 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.665172 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 02:41:50.665771 23323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
I20260620 02:41:50.667658 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.667233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38252)
0620 02:41:50.667430 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.667513 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:41:50.667977 23373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 02:41:50.672930 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
/tmp/dist-test-taskRSL2lK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34063
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37015,127.0.97.61:34909,127.0.97.60:34063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:41:50.699287 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } attempt: 5
Jun 20 02:41:50 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:41:50.716952 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.700353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.0.97.61:34909)
0620 02:41:50.700562 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:50.700569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:50.700592 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:41:50.700657 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:50.700662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:50.700767 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:50.700877 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:50.701729 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.702839 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.704269 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.704724 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.704875 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:50.708964 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:50.708973 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:50.708978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:50.709043 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:50.713153 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:50.714140 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:50.714145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:50.714148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:50.714291 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:50.714909 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:50.714916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:50.714919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:50.715657 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:50.715986 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:50.716374 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 02:41:50.716583 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:41:50.717000 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.700144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:43174)
0620 02:41:50.700370 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.700394 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:50.700526 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:50.700918 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:50.700921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:50.701330 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:50.701533 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:50.701543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.703022 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.703027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:50.704085 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:50.704100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:50.704891 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:50.704897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:50.705124 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:50.705841 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:50.705861 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:50.708453 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:50.713374 (+ 4921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:50.713381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:50.713384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:50.713964 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:50.714419 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:50.714423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:50.714427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:50.714611 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:50.715898 (+ 1287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:50.715903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:50.716102 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 02:41:50.716268 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 02:41:50.716627 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 02:41:50.717366 23373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } attempt: 1
I20260620 02:41:50.718703 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:41:50.718890 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.717900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38268)
0620 02:41:50.718077 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.718188 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 02:41:50.720588 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.719901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38278)
0620 02:41:50.720137 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.720350 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 02:41:50.720624 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.720113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38288)
0620 02:41:50.720325 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.720465 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 02:41:50.721107 23373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
I20260620 02:41:50.721853 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.721413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38298)
0620 02:41:50.721609 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.721699 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 02:41:50.722175 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
I20260620 02:41:50.773448 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } attempt: 1
I20260620 02:41:50.774785 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.774258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38308)
0620 02:41:50.774456 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.774605 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 02:41:50.776292 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.775775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38312)
0620 02:41:50.776079 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.776160 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
W20260620 02:41:50.776722 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
I20260620 02:41:50.789381 23373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } attempt: 2
I20260620 02:41:50.790684 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.790180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38320)
0620 02:41:50.790409 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.790499 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 02:41:50.792255 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.791881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38326)
0620 02:41:50.792035 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.792122 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:41:50.792598 23373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
I20260620 02:41:50.845234 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } attempt: 2
I20260620 02:41:50.848121 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.846390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38330)
0620 02:41:50.847773 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.847918 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1175}
I20260620 02:41:50.849426 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.848952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38344)
0620 02:41:50.849192 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.849279 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260620 02:41:50.849766 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
I20260620 02:41:50.883901 23373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } attempt: 3
I20260620 02:41:50.885416 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.884821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38356)
0620 02:41:50.885124 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.885215 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 02:41:50.886941 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.886391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38358)
0620 02:41:50.886658 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.886769 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260620 02:41:50.887199 23373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
I20260620 02:41:50.956034 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } attempt: 3
I20260620 02:41:50.957443 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.956925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38362)
0620 02:41:50.957149 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.957256 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 02:41:50.958803 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:50.958364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38366)
0620 02:41:50.958600 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:50.958676 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 02:41:50.959091 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
I20260620 02:41:51.030484 23373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } attempt: 4
I20260620 02:41:51.032372 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.031582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38378)
0620 02:41:51.031970 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:51.032131 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 02:41:51.033802 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.033341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38380)
0620 02:41:51.033542 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:51.033637 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 02:41:51.034194 23373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
W20260620 02:41:51.073231 23381 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:41:51.073643 23381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:41:51.073712 23381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:41:51.073793 23381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:41:51.082895 23381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:41:51.083019 23381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:41:51.083079 23381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:41:51.083178 23381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:41:51.083290 23381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:41:51.094732 23381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37015,127.0.97.61:34909,127.0.97.60:34063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 00:50:33 UTC on ccycloud.quasar-rcadai.root.comops.site
build id 4456863
ASAN enabled
I20260620 02:41:51.096271 23381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:41:51.097710 23381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:41:51.113196 23386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:51.113235 23389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:41:51.113857 23387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:41:51.116240 23381 server_base.cc:1034] running on GCE node
I20260620 02:41:51.117473 23381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:41:51.119623 23381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:41:51.123238 23381 hybrid_clock.cc:648] HybridClock initialized: now 1781923311122976 us; error 163 us; skew 500 ppm
I20260620 02:41:51.124017 23381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:41:51 dist-test-slave-rpg6 krb5kdc[23219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:41:51.130194 23381 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 02:41:51.132319 23381 webserver.cc:460] Webserver started at http://127.0.97.60:44809/ using document root <none> and password file <none>
I20260620 02:41:51.133163 23381 fs_manager.cc:359] Metadata directory not provided
I20260620 02:41:51.133320 23381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:41:51.133673 23381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:41:51.136385 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } attempt: 4
I20260620 02:41:51.138206 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.137525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38392)
0620 02:41:51.137913 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:51.138034 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260620 02:41:51.138049 23381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/data/instance:
uuid: "817b1061692d40078a0ddb8313193613"
format_stamp: "Formatted at 2026-06-20 02:41:51 on dist-test-slave-rpg6"
server_key: "4c8c28d31c0b103c8ced5bc3acc67b41"
server_key_iv: "8dd6d142c394eebb98d919494b18c2b9"
server_key_version: "encryptionkey@0"
I20260620 02:41:51.139151 23381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/wal/instance:
uuid: "817b1061692d40078a0ddb8313193613"
format_stamp: "Formatted at 2026-06-20 02:41:51 on dist-test-slave-rpg6"
server_key: "4c8c28d31c0b103c8ced5bc3acc67b41"
server_key_iv: "8dd6d142c394eebb98d919494b18c2b9"
server_key_version: "encryptionkey@0"
I20260620 02:41:51.139912 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.139205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:38398)
0620 02:41:51.139599 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:51.139720 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
W20260620 02:41:51.140211 23299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34063: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34063: connect: Connection refused (error 111)
I20260620 02:41:51.146057 23381 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:41:51.150394 23396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:41:51.152113 23381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:41:51.152333 23381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/wal
uuid: "817b1061692d40078a0ddb8313193613"
format_stamp: "Formatted at 2026-06-20 02:41:51 on dist-test-slave-rpg6"
server_key: "4c8c28d31c0b103c8ced5bc3acc67b41"
server_key_iv: "8dd6d142c394eebb98d919494b18c2b9"
server_key_version: "encryptionkey@0"
I20260620 02:41:51.152570 23381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:41:51.171670 23381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:41:51.176699 23381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:41:51.177013 23381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:41:51.198452 23381 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34063
I20260620 02:41:51.198473 23447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34063 every 8 connection(s)
I20260620 02:41:51.200137 23381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:41:51.204411 23448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:41:51.204535 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu as pid 23381
I20260620 02:41:51.204789 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRSL2lK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781922915737669-388-0/minicluster-data/master-2/wal/instance
I20260620 02:41:51.205310 388 external_mini_cluster.cc:1428] Setting key 66a602f936213a16a6c771e986ec516b
I20260620 02:41:51.215085 23448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:41:51 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:41:51 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:41:51.235041 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.207299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.0.97.60:34063)
0620 02:41:51.208262 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.208280 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.208340 (+ 60us) server_negotiation.cc:407] Connection header received
0620 02:41:51.210293 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.210335 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.210971 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.211507 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.213770 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.215800 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.217648 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.218875 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.219868 (+ 993us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.224891 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:51.224905 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:51.224927 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:51.225007 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:51.228939 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.229777 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.229784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.229793 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.229953 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.230430 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.230437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.230442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.231422 (+ 980us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:51.231668 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:51.232766 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0620 02:41:51.233344 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":440,"threads_started":1}
I20260620 02:41:51.237195 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.218594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37015 (local address 127.0.0.1:43978)
0620 02:41:51.219455 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:51.219493 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:51.219694 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:51.220235 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:51.220243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:51.220898 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:51.221149 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:51.221158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.222660 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.222672 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:51.223868 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:51.223896 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.224804 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.224815 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:51.225146 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.226178 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:51.226247 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:51.229151 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:51.233976 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:51.233987 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:51.233992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:51.234492 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:51.235038 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:51.235041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:51.235043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:51.235244 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:51.235947 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:51.235954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:51.236169 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 02:41:51.236383 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 02:41:51.236813 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":496,"threads_started":1}
I20260620 02:41:51.237341 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.218575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.0.97.62:37015)
0620 02:41:51.219149 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.219158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.219698 (+ 540us) server_negotiation.cc:407] Connection header received
0620 02:41:51.219889 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.219897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.220008 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.220119 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.221334 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.222451 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.224044 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.224497 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.224672 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.229446 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:51.229453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:51.229457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:51.229508 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:51.233764 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.234672 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.234679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.234683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.234891 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.235389 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.235395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.235398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.235777 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:51.235979 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:51.236525 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 02:41:51.236922 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":279,"threads_started":1}
I20260620 02:41:51.239028 23448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:41:51 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 02:41:51 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:41:51.256060 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.240024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34909 (local address 127.0.0.1:46044)
0620 02:41:51.240219 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:51.240240 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:51.240350 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:51.240841 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:51.240845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:51.241336 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:51.241530 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:51.241544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.242798 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.242804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:51.244333 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:51.244353 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.245112 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.245119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:51.245329 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.245985 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:51.246004 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:51.248912 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:51.253355 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:51.253363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:51.253374 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:51.253794 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:51.254245 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:51.254252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:51.254255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:51.254404 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:51.254976 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:51.254981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:51.255148 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 02:41:51.255299 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 02:41:51.255629 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 02:41:51.256124 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.240028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.0.97.61:34909)
0620 02:41:51.240521 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.240526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.240540 (+ 14us) server_negotiation.cc:407] Connection header received
0620 02:41:51.240587 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.240590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.240659 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.240744 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.241742 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.242576 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.244525 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.244940 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.245084 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.249437 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:51.249446 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:51.249450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:51.249510 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:51.253162 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.253961 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.253967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.253969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.254115 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.254532 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.254542 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.254544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.254836 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:51.255019 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:51.255425 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 02:41:51.255652 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":299,"threads_started":1}
I20260620 02:41:51.257174 23448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:41:51.259815 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.239242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.0.97.62:37015)
0620 02:41:51.239435 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.239440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.239458 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:41:51.239646 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.239651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.239739 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.239867 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.241651 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.242483 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.244442 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.244967 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.245190 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.252300 (+ 7110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:51.252306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:51.252310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:51.252355 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:51.255078 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.255965 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.255970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.255972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.256108 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.256762 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.256767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.256771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.257140 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:51.257545 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:51.259093 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0620 02:41:51.259478 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 02:41:51 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:41:51 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:41:51.275635 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.258241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:60348)
0620 02:41:51.258490 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:51.258512 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:51.258620 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:51.259078 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:51.259082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:51.259499 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:51.259711 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:51.259725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.261442 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.261450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:51.262756 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:51.262773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.263531 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.263539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:51.263785 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.264897 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:51.264935 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:51.268392 (+ 3457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:51.272565 (+ 4173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:51.272571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:51.272583 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:51.273109 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:51.273600 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:51.273604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:51.273607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:51.273831 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:51.274539 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:51.274543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:51.274678 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 02:41:51.274803 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 02:41:51.275266 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 02:41:51.276023 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.258454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.0.97.60:34063)
0620 02:41:51.258609 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.258614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.258631 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:41:51.258877 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.258882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.258962 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.259065 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.259928 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.261318 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.262931 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.263358 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.263477 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.268648 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:51.268654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:51.268657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:51.268691 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:51.272383 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.273281 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.273288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.273291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.273442 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.273962 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.273970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.273976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.274376 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:51.274542 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:51.274939 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 02:41:51.275582 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":309,"server-negotiator.queue_time_us":68}
I20260620 02:41:51.280491 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.262195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.0.97.61:34909)
0620 02:41:51.262381 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.262385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.262402 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:41:51.262451 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.262455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.262514 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.262688 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.264324 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.265043 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.267177 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.267891 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.268031 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.273368 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:51.273379 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:51.273384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:51.273430 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:51.277348 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.278325 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.278330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.278333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.278484 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.278941 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.278944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.278946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.279265 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:51.279537 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:51.280009 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 02:41:51.280240 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:41:51.282132 23448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613: Bootstrap starting.
I20260620 02:41:51.286568 23448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:51.286947 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:41:51.288107 23448 log.cc:824] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:51.291980 23448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613: No bootstrap required, opened a new log
I20260620 02:41:51.290284 23274 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "623926bf87cc43db8ab34fd2f01f4343"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39824
I20260620 02:41:51.304227 23448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } }
I20260620 02:41:51.304765 23448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817b1061692d40078a0ddb8313193613, State: Initialized, Role: FOLLOWER
I20260620 02:41:51.305320 23448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } }
I20260620 02:41:51.306409 23456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } } }
I20260620 02:41:51.306793 23456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:51.307880 23448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:41:51.319937 23467 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:41:51.320127 23467 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:41:51.317206 23381 master_runner.cc:186] Error getting master registration for 127.0.97.62:37015: OK, instance_id { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" instance_seqno: 1781923309968141 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:41:51.326709 23381 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:41:51.333513 23373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } attempt: 5
Jun 20 02:41:51 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:41:51.352054 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.334467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:60356)
0620 02:41:51.334726 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:51.334751 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:51.334915 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:51.335351 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:51.335357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:51.336098 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:51.336318 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:51.336330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.337909 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.337914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:51.339352 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:51.339373 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.340685 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.340691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:51.340934 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.341722 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:51.341747 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:51.344340 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:51.348770 (+ 4430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:51.348776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:51.348781 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:51.349384 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:51.349891 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:51.349895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:51.349900 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:51.350078 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:51.350926 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:51.350931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:51.351100 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 02:41:51.351247 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 02:41:51.351585 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 02:41:51.352058 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.334645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.0.97.60:34063)
0620 02:41:51.334854 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.334861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.334886 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:41:51.335074 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.335080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.335169 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.335296 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.336498 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.337717 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.339545 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.340497 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.340648 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.344587 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:51.344595 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:51.344602 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:51.344650 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:51.348546 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.349585 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.349592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.349595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.349759 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.350207 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.350213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.350216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.350697 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:51.350998 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:51.351373 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 02:41:51.351632 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 02:41:51.358706 23373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506: Bootstrap starting.
I20260620 02:41:51.363476 23373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:51.365211 23373 log.cc:824] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:51.369331 23373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506: No bootstrap required, opened a new log
I20260620 02:41:51.381211 23373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } }
I20260620 02:41:51.381673 23373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2adf95ccf4d2bbb4853ebce43b506, State: Initialized, Role: FOLLOWER
I20260620 02:41:51.382273 23373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } }
I20260620 02:41:51.383386 23470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } } }
I20260620 02:41:51.383888 23470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:51.384859 23373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:41:51.397801 23481 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:41:51.398094 23481 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:41:51.394338 23306 master_runner.cc:186] Error getting master registration for 127.0.97.62:37015: OK, instance_id { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" instance_seqno: 1781923309968141 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:41:51.407361 23306 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:41:51.417510 23299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } attempt: 5
Jun 20 02:41:51 dist-test-slave-rpg6 krb5kdc[23219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260620 02:41:51.429838 23402 tablet.cc:2369] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:41:51.438879 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.418405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34063 (local address 127.0.0.1:60370)
0620 02:41:51.418636 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 02:41:51.418659 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:41:51.418811 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:41:51.419867 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:41:51.419873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:41:51.420426 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:41:51.420665 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:51.420680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.422212 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.422218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:51.423681 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:41:51.423700 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.425071 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.425078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:41:51.425308 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.426121 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:41:51.426147 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:41:51.429371 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:41:51.435565 (+ 6194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:51.435575 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:51.435580 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:51.436235 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:51.436760 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:41:51.436766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:41:51.436770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:41:51.436984 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:41:51.437701 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:41:51.437707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:41:51.437923 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 02:41:51.438079 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 02:41:51.438406 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 02:41:51.438882 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.418973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60370 (local address 127.0.97.60:34063)
0620 02:41:51.419292 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.419353 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.419376 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:41:51.419442 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.419448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.419535 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.419640 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.420869 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.422010 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.423929 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.424693 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.424833 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.430986 (+ 6153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:51.430995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:51.431000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:51.431048 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:51.435354 (+ 4306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.436411 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.436418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.436422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.436619 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.437126 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.437132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.437136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.437553 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:51.437718 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:51.438236 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 02:41:51.438497 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 02:41:51.446038 23299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343: Bootstrap starting.
I20260620 02:41:51.450770 23299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343: Neither blocks nor log segments found. Creating new log.
I20260620 02:41:51.452440 23299 log.cc:824] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343: Log is configured to *not* fsync() on all Append() calls
I20260620 02:41:51.462850 23299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343: No bootstrap required, opened a new log
I20260620 02:41:51.472425 23299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } }
I20260620 02:41:51.472930 23299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623926bf87cc43db8ab34fd2f01f4343, State: Initialized, Role: FOLLOWER
I20260620 02:41:51.473662 23299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } }
I20260620 02:41:51.475220 23484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } } }
I20260620 02:41:51.476068 23484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:51.476490 23299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:41:51.476562 23274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 02:41:51.476773 23274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:51.480515 23274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } }
I20260620 02:41:51.483235 23348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "623926bf87cc43db8ab34fd2f01f4343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506"
I20260620 02:41:51.483793 23348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:51.488459 23348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623926bf87cc43db8ab34fd2f01f4343 in term 1.
I20260620 02:41:51.489562 23248 leader_election.cc:302] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ed2adf95ccf4d2bbb4853ebce43b506, 623926bf87cc43db8ab34fd2f01f4343; no voters:
I20260620 02:41:51.491679 23423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "623926bf87cc43db8ab34fd2f01f4343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "817b1061692d40078a0ddb8313193613"
I20260620 02:41:51.492303 23423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:41:51.496553 23423 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623926bf87cc43db8ab34fd2f01f4343 in term 1.
I20260620 02:41:51.497563 23274 leader_election.cc:288] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [CANDIDATE]: Term 1 election: Requested vote from peers 4ed2adf95ccf4d2bbb4853ebce43b506 (127.0.97.61:34909), 817b1061692d40078a0ddb8313193613 (127.0.97.60:34063)
I20260620 02:41:51.504870 23484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:41:51.515202 23484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [term 1 LEADER]: Becoming Leader. State: Replica: 623926bf87cc43db8ab34fd2f01f4343, State: Running, Role: LEADER
I20260620 02:41:51.516537 23484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } }
I20260620 02:41:51.528277 23490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 623926bf87cc43db8ab34fd2f01f4343. Latest consensus state: current_term: 1 leader_uuid: "623926bf87cc43db8ab34fd2f01f4343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } } }
I20260620 02:41:51.528950 23490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:51.533917 23497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260620 02:41:51.573998 23327 tablet.cc:2369] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:41:51.577716 23497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:41:51.586279 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.540174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.0.97.62:37015)
0620 02:41:51.540385 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.540391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.555287 (+ 14896us) server_negotiation.cc:407] Connection header received
0620 02:41:51.555528 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.555534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.555726 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.555891 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.558327 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.559505 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.561713 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.562508 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.562636 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.569148 (+ 6512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:51.569156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:51.569162 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:51.569249 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:51.573603 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.576035 (+ 2432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.576043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.576047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.576325 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.577164 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.577170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.577174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.577674 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:51.578616 (+ 942us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:51.585494 (+ 6878us) server_negotiation.cc:299] Negotiation successful
0620 02:41:51.585834 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:41:51.591115 23506 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:41:51.611368 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.541390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.0.97.61:34909)
0620 02:41:51.541648 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.541658 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.559488 (+ 17830us) server_negotiation.cc:407] Connection header received
0620 02:41:51.568296 (+ 8808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.568304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.568415 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.568500 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.571853 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.573101 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.584311 (+ 11210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.585036 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.585164 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.598733 (+ 13569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:51.598741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:51.598764 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:51.598866 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:51.603261 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.604847 (+ 1586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.604855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.604859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.605034 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.607685 (+ 2651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.607693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.607698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.608197 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:51.608500 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:51.610635 (+ 2135us) server_negotiation.cc:299] Negotiation successful
0620 02:41:51.610922 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:41:51.637228 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:51.571114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.0.97.60:34063)
0620 02:41:51.571349 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.571355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.576176 (+ 4821us) server_negotiation.cc:407] Connection header received
0620 02:41:51.586528 (+ 10352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.586538 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.586650 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.586745 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.595703 (+ 8958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.596904 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.605558 (+ 8654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:51.606403 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:51.606526 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:51.619657 (+ 13131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:41:51.619667 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:41:51.619682 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:41:51.619755 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:41:51.624135 (+ 4380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.632234 (+ 8099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.632286 (+ 52us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.632293 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.632958 (+ 665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:41:51.634155 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:41:51.634162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:41:51.634165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:41:51.634738 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:41:51.635096 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:51.636290 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0620 02:41:51.636718 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:41:51.641424 23423 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [term 1 FOLLOWER]: Refusing update from remote peer 623926bf87cc43db8ab34fd2f01f4343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:41:51.641424 23348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [term 1 FOLLOWER]: Refusing update from remote peer 623926bf87cc43db8ab34fd2f01f4343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:41:51.642987 23484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:41:51.644548 23484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:41:51.670770 23470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 623926bf87cc43db8ab34fd2f01f4343. Latest consensus state: current_term: 1 leader_uuid: "623926bf87cc43db8ab34fd2f01f4343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } } }
I20260620 02:41:51.671119 23470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:51.680603 23456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 623926bf87cc43db8ab34fd2f01f4343. Latest consensus state: current_term: 1 leader_uuid: "623926bf87cc43db8ab34fd2f01f4343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } } }
I20260620 02:41:51.681080 23456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:51.682022 23508 mvcc.cc:205] Tried to move back new op lower bound from 7298757884464177152 to 7298757883992768512. Current Snapshot: MvccSnapshot[applied={T|T < 7298757884464177152}]
I20260620 02:41:51.686657 23470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "623926bf87cc43db8ab34fd2f01f4343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } } }
I20260620 02:41:51.686967 23470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:51.700538 23456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "623926bf87cc43db8ab34fd2f01f4343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } } }
I20260620 02:41:51.700856 23456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:41:51.704031 23497 catalog_manager.cc:1347] Generated new cluster ID: 34fc1bb2dafc4f1fbe098a8b19c9edbe
I20260620 02:41:51.704171 23497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:41:51.704422 23484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "623926bf87cc43db8ab34fd2f01f4343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } } }
I20260620 02:41:51.704787 23484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:51.705071 23484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "623926bf87cc43db8ab34fd2f01f4343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623926bf87cc43db8ab34fd2f01f4343" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } peers { permanent_uuid: "4ed2adf95ccf4d2bbb4853ebce43b506" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34909 } } peers { permanent_uuid: "817b1061692d40078a0ddb8313193613" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34063 } } }
I20260620 02:41:51.705360 23484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343 [sys.catalog]: This master's current role is: LEADER
I20260620 02:41:51.743592 23497 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:41:51.745724 23497 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:41:51.772845 23497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 623926bf87cc43db8ab34fd2f01f4343: Generated new TSK 0
I20260620 02:41:51.774463 23497 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:41:51.907114 23452 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:51.894482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.0.97.62:37015)
0620 02:41:51.894748 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.894754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.904614 (+ 9860us) server_negotiation.cc:407] Connection header received
0620 02:41:51.904981 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.904992 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.905115 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.905225 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.906840 (+ 1615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39844: BlockingRecv error: recv got EOF from 127.0.0.1:39844 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 02:41:51.910234 23454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:51.899614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.0.97.61:34909)
0620 02:41:51.899894 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.899901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.907978 (+ 8077us) server_negotiation.cc:407] Connection header received
0620 02:41:51.908331 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.908337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.908426 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.908525 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.909863 (+ 1338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46072: BlockingRecv error: recv got EOF from 127.0.0.1:46072 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 02:41:51.913039 23451 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:51.908000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.0.97.60:34063)
0620 02:41:51.908210 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.908216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.910868 (+ 2652us) server_negotiation.cc:407] Connection header received
0620 02:41:51.911180 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.911186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.911308 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.911448 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.912739 (+ 1291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60376: BlockingRecv error: recv got EOF from 127.0.0.1:60376 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 02:41:51.951924 23454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:51.938479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.0.97.61:34909)
0620 02:41:51.938737 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.938744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.949559 (+ 10815us) server_negotiation.cc:407] Connection header received
0620 02:41:51.950184 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.950191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.950334 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.950430 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:51.951604 (+ 1174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46086: BlockingRecv error: recv got EOF from 127.0.0.1:46086 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 02:41:52.000818 23454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:51.989156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.0.97.61:34909)
0620 02:41:51.989352 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:51.989358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:51.998641 (+ 9283us) server_negotiation.cc:407] Connection header received
0620 02:41:51.999214 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:51.999222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:51.999329 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:51.999407 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:52.000555 (+ 1148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46100: BlockingRecv error: recv got EOF from 127.0.0.1:46100 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 02:41:52.040910 23454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:52.038207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46110 (local address 127.0.97.61:34909)
0620 02:41:52.038408 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.038415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.038989 (+ 574us) server_negotiation.cc:407] Connection header received
0620 02:41:52.039421 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.039427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.039541 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.039703 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:52.040620 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46110: BlockingRecv error: recv got EOF from 127.0.0.1:46110 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 02:41:52.052310 23454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:52.049821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.0.97.61:34909)
0620 02:41:52.050106 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.050112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.050714 (+ 602us) server_negotiation.cc:407] Connection header received
0620 02:41:52.050918 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.050925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.051025 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.051134 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:52.052065 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46126: BlockingRecv error: recv got EOF from 127.0.0.1:46126 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260620 02:41:52.069681 23454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:52.066983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.0.97.61:34909)
0620 02:41:52.067266 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.067272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.067928 (+ 656us) server_negotiation.cc:407] Connection header received
0620 02:41:52.068162 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.068168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.068277 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.068405 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:52.069383 (+ 978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46142: BlockingRecv error: recv got EOF from 127.0.0.1:46142 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
W20260620 02:41:52.093739 23454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:52.090714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.0.97.61:34909)
0620 02:41:52.090911 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.090918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.091551 (+ 633us) server_negotiation.cc:407] Connection header received
0620 02:41:52.091722 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.091729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.091861 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.091984 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:52.093471 (+ 1487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46148: BlockingRecv error: recv got EOF from 127.0.0.1:46148 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 02:41:52.132231 23454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:52.129970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.0.97.61:34909)
0620 02:41:52.130205 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.130211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.130500 (+ 289us) server_negotiation.cc:407] Connection header received
0620 02:41:52.130668 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.130674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.130794 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.130929 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:52.131958 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46152: BlockingRecv error: recv got EOF from 127.0.0.1:46152 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 02:41:52.203518 23454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:52.201695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.0.97.61:34909)
0620 02:41:52.201904 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.201912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.202056 (+ 144us) server_negotiation.cc:407] Connection header received
0620 02:41:52.202320 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.202325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.202402 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.202518 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:52.203280 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46160: BlockingRecv error: recv got EOF from 127.0.0.1:46160 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:41:52.322321 23467 catalog_manager.cc:1259] Loaded cluster ID: 34fc1bb2dafc4f1fbe098a8b19c9edbe
I20260620 02:41:52.322438 23467 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613: loading cluster ID for follower catalog manager: success
I20260620 02:41:52.325331 23467 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613: acquiring CA information for follower catalog manager: success
I20260620 02:41:52.327593 23467 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 817b1061692d40078a0ddb8313193613: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 02:41:52.339170 23454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:41:52.335726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.0.97.61:34909)
0620 02:41:52.336046 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.336053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.336348 (+ 295us) server_negotiation.cc:407] Connection header received
0620 02:41:52.336479 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.336484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.336579 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.336708 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:41:52.338787 (+ 2079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46162: BlockingRecv error: recv got EOF from 127.0.0.1:46162 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
I20260620 02:41:52.400522 23481 catalog_manager.cc:1259] Loaded cluster ID: 34fc1bb2dafc4f1fbe098a8b19c9edbe
I20260620 02:41:52.400653 23481 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506: loading cluster ID for follower catalog manager: success
I20260620 02:41:52.403509 23481 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506: acquiring CA information for follower catalog manager: success
I20260620 02:41:52.405813 23481 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ed2adf95ccf4d2bbb4853ebce43b506: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:41:52.613143 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:52.605488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.0.97.61:34909)
0620 02:41:52.605659 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.605664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.606129 (+ 465us) server_negotiation.cc:407] Connection header received
0620 02:41:52.606307 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.606312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.606556 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.606690 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:41:52.607192 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:52.608589 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:52.610585 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:52.611021 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:52.611138 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:52.611800 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:41:52.612436 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:41:52.612612 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:52.612754 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 02:41:52.612854 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 02:41:52.629236 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:52.621159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.0.97.60:34063)
0620 02:41:52.621831 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.621836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.621857 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:41:52.621960 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.621964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.622148 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.622315 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:41:52.622853 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:52.623748 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:52.625449 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:52.625962 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:52.626075 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:52.627161 (+ 1086us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:41:52.628258 (+ 1097us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:41:52.628475 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:52.628726 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 02:41:52.628864 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":316,"threads_started":1}
I20260620 02:41:52.647560 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:52.638075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.0.97.61:34909)
0620 02:41:52.638280 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.638290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.639938 (+ 1648us) server_negotiation.cc:407] Connection header received
0620 02:41:52.640268 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.640276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.640426 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.640501 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:41:52.641683 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:52.642887 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:52.645516 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:52.645998 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:52.646167 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:52.646746 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:41:52.646899 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:41:52.646991 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:52.647168 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 02:41:52.647258 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:41:52.649564 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:52.639405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.0.97.60:34063)
0620 02:41:52.639632 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.639752 (+ 120us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.641444 (+ 1692us) server_negotiation.cc:407] Connection header received
0620 02:41:52.641534 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.641540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.641716 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.641819 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:41:52.642491 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:52.643989 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:52.647092 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:52.647964 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:52.648113 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:52.648543 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:41:52.648679 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:41:52.648775 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:52.648924 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 02:41:52.649293 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:41:52.650416 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:41:52.638074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.0.97.62:37015)
0620 02:41:52.640509 (+ 2435us) server_negotiation.cc:206] Beginning negotiation
0620 02:41:52.640520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:41:52.640557 (+ 37us) server_negotiation.cc:407] Connection header received
0620 02:41:52.640763 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:41:52.640769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:41:52.641210 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:41:52.641460 (+ 250us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:41:52.642183 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:52.643180 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:52.646882 (+ 3702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:41:52.647498 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:41:52.647655 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:41:52.649278 (+ 1623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:41:52.649782 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:41:52.649928 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 02:41:52.650030 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 02:41:52.650123 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2263,"thread_start_us":2073,"threads_started":1}
I20260620 02:41:52.657645 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 23232
I20260620 02:41:52.658972 23294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:52.778589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 23232
I20260620 02:41:52.802336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 23306
I20260620 02:41:52.804067 23368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:52.925361 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 23306
I20260620 02:41:52.947289 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu pid 23381
I20260620 02:41:52.949247 23443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:41:53.066614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRSL2lK/build/asan/bin/kudu with pid 23381
2026-06-20T02:41:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5135 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10156 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (397289 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST